Sandfly 2.3 – Performance Updates, Elasticsearch 7 Support and More

Product Update

Date
October 29, 2019
Author
The Sandfly Security Team

Sandfly 2.3 is now released. This version features many changes to improve performance, updates the custom Sandfly syntax, eliminates false alarms and migrates to Elasticsearch 7.

Performance Enhancements

The backend has been made significantly faster and more robust with internal database optimizations. Additionally, host investigations have been made faster with less RAM and CPU usage by optimizing the forensic engines.

Sandfly 2.3 Syntax Updates

Changes to the syntax used for writing Sandfly modules have been made. We have added more and more forensic investigation features as our product has matured. As a result, the number of keywords and how they were organized began to grow and we wanted to change the structure to keep them under control as we enhance our product. In Sandfly 2.3 we made changes to how the Sandfly JSON syntax is handled to make it cleaner and easier to expand going forward.

If you have existing custom Sandfly modules, they will be automatically converted to the new format during the update. You shouldn’t have to do anything to use the new syntax.

Forensic Data Updates

We have improved the forensic data keywords to handle the growing list of threats we can investigate. The new format will allow queries for SIEM integration to be clearer and is also easier for operators to read. The arrows below show the major changes to the forensic data format:

New JSON format example

Date Format

Dates are now represented under their own key. All forensic artifacts whether for a process, file, directory, user or log entry will have a similar format to that above.

Cryptographic Hashes Format

Like dates, we have broken out all cryptographic hashes into their own key. Under this key you will find the MD5, SHA1, SHA256 and SHA512 hashes where available (such as for files and process binaries).

File Magic Number Format

Magic number data on files have likewise been moved to their own key for easier grouping.

File, Directory and Process Flags

We also expanded and made a new grouping for file, directory and process data flags. We also added a couple new flags and again grouped it to be clearer.

Scanning Performance and Reliability Updates

We have changed the Sandfly checks to improve our ability to gracefully stop a scan if the remote system is experiencing trouble. We can now apply individual timeouts to each check more precisely and reliably. We will halt a scan entirely if we get multiple timeouts to prevent overloading a system that may be having load issues independent of Sandfly.

Duplicate Scanning Protection

Duplicate scans are no longer allowed. In previous versions it was possible to kick off two or more scans at once on a host. Under some situations this can cause Sandfly to false alarm. We no longer allow this situation to happen.

If a host is shown in the scan queue, Sandfly will not allow you to scan it again until it completes. More

Elasticsearch 7 Updates

We have moved from Elasticsearch 6 to Elasticsearch 7 on the backend. Mostly this will be invisible to customers, however we will purge all old result sets during the upgrade to be sure the new data types required by Elasticsearch 7 are done correctly. We also will purge the scheduled tasks as the older timestamp format used internally has been updated. You will need to add your scheduled scans back in after this update.

Malicious Scanning Activity Detection and More

We have expanded our ability to better categorize network connections to and from protected hosts. The new network features mean we can track open ports, established ports and listening ports. We can do this for TCP, UDP, TCPv6, UDPv6, ICMP and also RAW network ports and protocols. One feature to come out of this is our ability to detect processes that have many outbound network connections which is often associated with malware. It can also help spot users that may be running port scanners and similar malicious activity from a system.

Detecting malicious port scanning on Linux.

New Setup Scripts With Version

We are moving away from using the latest tag in our Docker images. The latest tag can cause confusion and we’d rather users be very exact about what version they are using. As a result, we are getting rid of using latest and using the version number directly. This change means customers will need to use the most up to date scripts to get the latest versions. This ensures you will be using the version you expect if you ever needed to install Sandfly on other systems.

What this means is that setup scripts will need to be updated. Be sure you do a git pull inside the setup directory before doing your upgrade. After this release, the latest tag will be deleted and older setup scripts will not work.

Bug Fixes and False Alarms

We have made bug and false alarm fixes internally. Some of these revolve around improved methods for our forensic engines, plus new ways to search for malware we found during R&D efforts. We dislike false alarms and appreciate customers that report them to us so we can investigate and remedy them.

How to Upgrade Sandfly

Sandfly is easy to upgrade. Please follow the instructions here:

Upgrading Sandfly

Nearly 650 Sandfly Checks and Growing

Sandfly 2.3 has now brought the number of compromise and incident response checks we do on Linux up to nearly 650. We can spot a tremendous amount of Linux malware, rootkits and intruder activity without loading any agents on your endpoints. Thank you for using our product.


Let Sandfly keep your Linux systems secure.

Learn More