How do I set up a mail server on a VPS and why do your emails end up in spam?

Giteqa

Greetings, friends!

Every entrepreneur or system administrator eventually faces the necessity of sending emails from a server. These could be transactional notifications from an e-commerce store, password reset links from a CRM, or standard corporate correspondence. The simplest route is to connect a third-party SMTP gateway or a corporate package from tech giants. However, continuous price increases, strict daily email limits, and the risk of sudden account suspension force businesses to look toward their own mail server on a VPS.

And this is where beginners encounter the main surprise: setting up the server itself (such as Postfix or Exim) takes just 10 minutes, but 99% of sent emails will silently land in the "Spam" folder or get blocked completely by email providers (Gmail, Outlook).

In 2026, spam filters have become paranoidly strict. In this guide, we will break down in detail how modern email network etiquette works, why major services do not trust your IP address, and how to configure your server step-by-step so that emails always reach the "Inbox."

Key Takeaways: Secrets of Successful Email Delivery

  • IP Reputation is 80% of success: If someone engaged in aggressive spamming on this IP address before you, your emails will be blocked regardless of how ideal your configurations are.

  • The Holy Trinity of DNS (SPF, DKIM, DMARC): Without these three digital signatures in your domain's DNS zone, modern email providers won't even consider reviewing your email.

  • rDNS (PTR Record): Your mail server's name must strictly match the reverse DNS record of your IP address. This is the primary legitimacy marker for spam filters.

  • IP Warming: You cannot instantly send a newsletter to 10,000 addresses from a brand-new server—spam systems will view this as an anomaly and ban the entire subnet.

How to Install a Mail Server?

We have recorded a guide that will show you how to install a very simple-to-use and easy-to-configure mail server, namely Listmonk:

You will find all the setup commands in the video description and the pinned comment. Even I was able to install, configure, and launch a newsletter to a very large database independently while avoiding IP blocks and spam folders across all email providers.

Step-by-Step Mail Server Configuration Guide

To minimize risks and guarantee maximum deliverability, infrastructure installation and configuration must be performed in a strict sequential order.

1. IP Address Verification and rDNS (PTR Record) Setup (2-3 minutes)

Before installing any software, check your IP for blocklist entries via services like MXToolbox. If the IP is clean, log in to your hosting control panel and set up a PTR record. For example, if your server is named mail.yourdomain.com, querying the reverse DNS of your IP must return this exact address. If your IP happens to be listed, any reliable hosting provider will swap it for a clean one upon your request.

2. Deploying the Mail Stack on Ubuntu 24.04 (5-10 minutes)

To avoid configuring Postfix, Dovecot, and a DBMS manually from scratch, utilize verified open-source Docker solutions (such as docker-mailserver) or auto-installation scripts (like Mail-in-a-Box). This eliminates human error and basic security loopholes, saving you a tremendous amount of time.

3. Configuring Cryptographic Records in DNS (5 minutes)

Add three mandatory records to your registrar's DNS panel:

  • SPF (TXT Record): Specifies exactly which IP addresses are permitted to send mail on behalf of your domain. Example: v=spf1 ip4:192.0.2.1 ~all.

  • DKIM (TXT Record): The public key. The server signs each outgoing email with a hidden digital signature, and the recipient verifies it via DNS.

  • DMARC (TXT Record): Instructions for the recipient on what to do with an email if it fails SPF or DKIM verification. Example: v=DMARC1; p=quarantine; (move suspicious emails to spam rather than dropping them completely during the initial phase).

Comparison Table: Self-Hosted Mail Server vs SaaS (Google / 365)

CriterionSelf-Hosted Mail Server on VPSCorporate Email (SaaS)IT Infrastructure & Budget Impact
Financial CostFixed VPS rental price (does not depend on the number of mailboxes).Per-user monthly pricing structure.For teams larger than 10 people, a private server saves hundreds of euros annually.
Sending LimitsBounded only by hardware capacity and common sense.Strict daily limits apply (e.g., up to 2000 emails per day).A private server is ideal for legitimate transactional notifications.
Data Privacy100%. Emails and log files reside exclusively on your storage drives.Moderate. Corporate bots scan text data for AI training models.Guarantees complete digital sovereignty for the business.
Setup ComplexityRequires DNS, rDNS configuration, and active reputation control.Works out of the box following domain verification.Demands basic system administration skills.

Sending Mail Requires a Professional Approach!

When planning to launch an email newsletter, you need to understand one essential thing: even if you are conducting completely legal marketing (NOT spam) and have set everything up yourself, that alone is not enough. To know for certain that emails will reach your users rather than dropping into spam, you need proper IP warming and an impeccable technical reputation. Building such an infrastructure from scratch independently is extremely difficult. This is exactly why businesses require the assistance of specialized services like SendGrid.

SendGrid allows you not only to route mail through their trusted servers but also provides detailed analytics to track campaign effectiveness easily. Naturally, you have to pay for the service—rates depend on the monthly volume of sent emails. Additionally, be prepared for a compliance review: SendGrid's security team typically asks to verify your contact list origin, your email templates, and the subject matter of your campaigns. This is how the platform protects itself and its other clients from spammers.

Ultimately, using SendGrid saves your nerves and helps clean your database of "dead" email addresses, preventing repeated deliveries to non-existent mailboxes.

Top 3 Reasons Why Your Emails Are Still Going to Spam

If you have configured SPF, DKIM, and PTR, but your deliverability is still lagging, verify these non-obvious factors:

  1. Missing MX Record: Inbound mail servers check whether your domain can receive return emails. If the MX record is missing in DNS or points to a non-existent host, outbound messages will be rejected.

  2. Poor Content and Stop Words: Spam filters evaluate the body of the email. Excessive use of block capitals in the subject line ("BUY RIGHT NOW!!!"), hidden tracking pixels, shortened links, and the absence of a mandatory unsubscribe mechanism (List-Unsubscribe in headers) instantly lower your internal trust rating (Spam Score).

  3. Cold IP Address: If the server is completely new, send emails in small batches (e.g., 50-100 emails per day), gradually scaling up the volume over several weeks. This allows spam evaluation algorithms to get accustomed to your network profile.

FAQ: Quick Summary

  • How can I test my mail server's configuration quality?

    Utilize the free Mail-tester.com service. It provides a single-use email address where you can route a test message from your server. The system will calculate a score from 1 to 10 and explicitly display errors in your SPF, DKIM, or content layout.

  • Is it necessary to encrypt email traffic?

    Absolutely. Configuring TLS encryption (STARTTLS) for server-to-server mail relays is a default requirement in 2026. Without it, emails will either be dropped or flagged in the recipient's UI with a warning open-padlock icon.

Conclusion

A self-hosted mail server is a powerful asset for your business's independence, completely lifting limits on correspondence volumes and guaranteeing data privacy. Yes, it requires care during the initial DNS infrastructure setup, but this investment of time pays off with stable operations and full control over your outbound mail.

In the world of email deliverability, the cleanliness of the network neighborhood plays a critical role. If a hosting provider handles its IP ranges poorly and allows spammers to block entire subnets, your mail will be rejected alongside your neighbors despite perfect SPF and DKIM records.

If you are currently searching for a dependable environment to deploy corporate or transactional mail services, explore our SSD KVM VPS / NVME VPS / Dedicated Server offerings at MivoCloud. We strictly monitor the reputation of our network blocks and provide clients with flexible, seamless control over PTR records (rDNS) directly within the hosting control panel. This ensures your newsletters can be managed with ease.

Article Author — Anatolie Cohaniuc