Sandfly 2.3 – Performance Updates, Elasticsearch 7 Support and More
October 30, 2019
Sandfly 2.3 is now released. This version features many changes to improve performance, updates the custom Sandfly syntax, eliminates false alarms and migrates to Elasticsearch 7.
The backend has been made significantly faster and more robust with internal database optimizations. Additionally, host investigations have been made faster with less RAM and CPU usage by optimizing the forensic engines.
Sandfly 2.3 Syntax Updates
Changes to the syntax used for writing Sandfly modules have been made. We have added more and more forensic investigation features as our product has matured. As a result, the number of keywords and how they were organized began to grow and we wanted to change the structure to keep them under control as we enhance our product. In Sandfly 2.3 we made changes to how the Sandfly JSON syntax is handled to make it cleaner and easier to expand going forward.
If you have existing custom Sandfly modules, they will be automatically converted to the new format during the update. You shouldn’t have to do anything to use the new syntax.
Forensic Data Updates
We have improved the forensic data keywords to handle the growing list of threats we can investigate. The new format will allow queries for SIEM integration to be clearer and is also easier for operators to read. The arrows below show the major changes to the forensic data format:
Dates are now represented under their own key. All forensic artifacts whether for a process, file, directory, user or log entry will have a similar format to that above.
Cryptographic Hashes Format
Like dates, we have broken out all cryptographic hashes into their own key. Under this key you will find the MD5, SHA1, SHA256 and SHA512 hashes where available (such as for files and process binaries).
File Magic Number Format
Magic number data on files have likewise been moved to their own key for easier grouping.
File, Directory and Process Flags
We also expanded and made a new grouping for file, directory and process data flags. We also added a couple new flags and again grouped it to be clearer.
Scanning Performance and Reliability Updates
We have changed the Sandfly checks to improve our ability to gracefully stop a scan if the remote system is experiencing trouble. We can now apply individual timeouts to each check more precisely and reliably. We will halt a scan entirely if we get multiple timeouts to prevent overloading a system that may be having load issues independent of Sandfly.
Duplicate Scanning Protection
Duplicate scans are no longer allowed. In previous versions it was possible to kick off two or more scans at once on a host. Under some situations this can cause Sandfly to false alarm. We no longer allow this situation to happen.
If a host is shown in the scan queue, Sandfly will not allow you to scan it again until it completes. More importantly, the scheduler will not put a new scan in the queue if one is already there. The next time the scan is scheduled it will be executed normally if the queue backlog for the host is resolved.
Elasticsearch 7 Updates
We have moved from Elasticsearch 6 to Elasticsearch 7 on the backend. Mostly this will be invisible to customers, however we will purge all old result sets during the upgrade to be sure the new data types required by Elasticsearch 7 are done correctly. We also will purge the scheduled tasks as the older timestamp format used internally has been updated. You will need to add your scheduled scans back in after this update.
Malicious Scanning Activity Detection and More
We have expanded our ability to better categorize network connections to and from protected hosts. The new network features mean we can track open ports, established ports and listening ports. We can do this for TCP, UDP, TCPv6, UDPv6, ICMP and also RAW network ports and protocols. One feature to come out of this is our ability to detect processes that have many outbound network connections which is often associated with malware. It can also help spot users that may be running port scanners and similar malicious activity from a system.
New Setup Scripts With Version
We are moving away from using the latest tag in our Docker images. The latest tag can cause confusion and we’d rather users be very exact about what version they are using. As a result, we are getting rid of using latest and using the version number directly. This change means customers will need to use the most up to date scripts to get the latest versions. This ensures you will be using the version you expect if you ever needed to install Sandfly on other systems.
What this means is that setup scripts will need to be updated. Be sure you do a git pull inside the setup directory before doing your upgrade. After this release, the latest tag will be deleted and older setup scripts will not work.
Bug Fixes and False Alarms
We have made bug and false alarm fixes internally. Some of these revolve around improved methods for our forensic engines, plus new ways to search for malware we found during R&D efforts. We dislike false alarms and appreciate customers that report them to us so we can investigate and remedy them.
How to Upgrade Sandfly
Sandfly is easy to upgrade. Please follow the instructions here:Upgrading Sandfly
Nearly 650 Sandfly Checks and Growing
Sandfly 2.3 has now brought the number of compromise and incident response checks we do on Linux up to nearly 650. We can spot a tremendous amount of Linux malware, rootkits and intruder activity without loading any agents on your endpoints. Thank you for using our product.