InsightIDR’s Log Search: Recent Enhancements and Upcoming Investments

Log data is critical to ensuring that you have full visibility into what’s going on across your environment. Alongside endpoint and network data, log data enables teams to detect malicious activity, prove compliance, and have better visibility across their environment.

Within InsightIDR, our Log Search takes every log of raw, collected data and automatically sorts them into Log Sets for you, so you can:

• Search logs for specific terms with a variety of Search Languages.
• Build your own query to group by a field or calculate specific items.
• View logs in Visual Search.
• Create tags and Alerts on your log data.
• Export data to share with stakeholders.

We recently (virtually) sat down with Mirela Smlatic, a Senior Product Manager for Detection and Response at Rapid7, to hear about enhancements and upcoming investments into InsightIDR’s Log Search capabilities. Read on for the highlights from our conversation and get a sneak peek at some of what’s to come with Log Search this year!

Q: We know that security analysts spend a ton of their time in search. Can you explain how InsightIDR’s approach to search and analyzing data might differ from what people are used to in traditional SIEM solutions?

Mirela: A differentiator for us is that we understand the pain of the customer. When we make updates to log search, or any part of the InsightIDR product, the customer comes first, and that’s how we drive our requirements. We’re constantly sourcing feedback from our users, and have a direct feedback loop from our own Managed Detection and Response (MDR) service team here at Rapid7, who uses our product daily in their investigations.

With InsightIDR, we want users to be able to easily decide how they want to approach Log Search—with or without the need to write RegEx. For those interested in advanced searches, we’re continually improving our LEQL language to make it more human readable and intuitive. For simple searches, we’re working to make searching faster and easier by enabling users to build queries without needing to know RegEx.

Q: What are some of the more recent search investments you think customers will be most excited about?

Mirela: We’ve quite a few exciting Log Search improvements and releases recently, but I’d say some of the highlights are:

• Multi-groupby search. Toward the end of 2020, we introduced LEQL multi-groupby to give customers a more detailed view into their log data by allowing them to group up to five fields in one single query, which reduces the number of queries they need to run in multiple tabs and makes their searches more efficient. We have continued to make improvements to multi-groupby over the past few months.
• LEQL improvements. We know that writing complex regex and long queries can result in partial or no results at all, which results in more time and resources needed to resolve incidents. To remedy this, we added a new LEQL IN function. Instead of querying by where(a=v1 OR a=v2 OR a=v3 OR a=v4 a=v5), the customer can query the data in a more readable and shorter way where(a IN [v1,v2,v3,v4,v5]). This makes it easier to search through long lists in log search, especially when searching for unknowns. It will also really help for more advanced use cases like data based threat hunting. We will continue to invest in LEQL enhancements as the year goes on to help improve the flexibility and ease of use in Search.

“Overall, InsightIDR saves countless hours of digging through logs to easily drill down into activities by device, user, or basically anything you can search for. The product continues to evolve and provide even more real-world value.” – Information Security Officer via Gartner Peer Insights

Q: Can you give customers a tease of some things they may have to look forward too soon with search?

Mirela: Our vision for Log Search in 2021 is easier, faster, and more scalable log search, enhanced dashboards and reporting, and overall a more unified user experience. I’ll highlight a couple of the releases we have coming soon:

• The ability to search by Log Set. To accelerate creation of cards, we initially focused on users selecting individual log sets, which represented the bulk of use cases that we saw from customers. But what we’ve seen, especially in the past 12 months as our customers’ environments have really continued to grow exponentially, is that customers want to go beyond static logs and be able to tailor charts and searches to full log sets.

With this new improvement, dashboard cards will automatically update to include any logs that are added to the same log set, saving teams time and making their searches easier and more scalable.

• Updated dashboards and reporting. We want to provide InsightIDR customers with a consolidated view of their critical data in a customizable yet easy to use way. To achieve this, we are soon releasing significant updates to our dashboards and reporting.

Soon, we’ll be rolling out an expanded Card Library for instant charts, an improved Card Builder experience, expanded data sources for dashboards beyond log data, an improved appearance for scheduled reports, and the ability to email reports to any email address for easy sharing. We hope that these enhancements will allow customers to more easily navigate and visualize their data, as well as make reporting easier to share across their organization.
We hope this look into the recent and ongoing enhancements into Log Search provides you with inspiration and insight into what you can do with your log data in InsightIDR! For the latest on Log Search and more, be sure to stay up-to-date on our release notes.