GRR logo

GRR Rapid Response is an incident response framework focused on remote live forensics.

It consists of a python client (agent) that is installed on target systems, and python server infrastructure that can manage and talk to clients.

The goal of GRR is to support forensics and investigations in a fast, scalable manner to allow analysts to quickly triage attacks and perform analysis remotely.

GRR on GitHub

GRR is open source (Apache License 2.0) and is developed on GitHub: github.com/google/grr

Contacts

Table of contents

What is GRR?

GRR Rapid Response is an incident response framework focused on remote live forensics.

The goal of GRR is to support forensics and investigations in a fast, scalable manner to allow analysts to quickly triage attacks and perform analysis remotely.

GRR consists of 2 parts: client and server.

GRR client is deployed on systems that one might want to investigate. On every such system, once deployed, GRR client periodically polls GRR frontend servers for work. “Work” means running a specific action: downloading file, listing a directory, etc.

GRR server infrastructure consists of several components (frontends, workers, UI servers) and provides web-based graphical user interface and an API endpoint that allows analysts to schedule actions on clients and view and process collected data.

Remote forensics at scale

GRR was built to run at scale so that analysts are capable of effectively collecting and processing data from large numbers of machines. GRR was built with following scenarios in mind:

  • Joe saw something weird, check his machine (p.s. Joe is on holiday in Cambodia and on 3G)

  • Forensically acquire 25 machines for analysis (p.s. they’re in 5 continents and none are Windows)

  • Tell me if this machine is compromised (while you’re at it, check 100,000 of them - i.e. “hunt” across the fleet)

GRR client features

  • Cross-platform support for Linux, OS X and Windows clients.

  • Live remote memory analysis using YARA library.

  • Powerful search and download capabilities for files and the Windows registry.

  • OS-level and raw file system access, using the SleuthKit (TSK).

  • Secure communication infrastructure designed for Internet deployment.

  • Detailed monitoring of client CPU, memory, IO usage and self-imposed limits.

GRR server features

  • Fully fledged response capabilities handling most incident response and forensics tasks.

  • Enterprise hunting (searching across a fleet of machines) support.

  • Fast and simple collection of hundreds of digital forensic artifacts.

  • AngularJS Web UI and RESTful JSON API with client libraries in Python, PowerShell and Go.

  • Powerful data export features supporting variety of formats and output plugins.

  • Fully scalable back-end capable of handling large deployments.

  • Automated scheduling for recurring tasks.

  • Asynchronous design allowing future task scheduling for clients, designed to work with a large fleet of laptops.

Quickstart (have GRR running in 5 minutes)

This page describes how to get GRR clients and server components up and running for the first time. If you have Docker installed, and just want to experiment with a container that already has GRR set up, you can follow the instructions given here. It takes ~2 minutes to download the image and initialize the server.

To start, install the GRR server deb as described here.

On successful installation, you should have an admin interface running on port 8000 by default. Open the Admin UI in a browser, navigate to Binaries -> Executables (details here) and download the installer you need.

Next, run the client installer on the target machine (can be the same machine as the one running the GRR server components) as root:

  • For Windows you will see a 64 bit installer. Run the installer as admin (it should load the UAC prompt if you are not admin). It should run silently and install the client to c:\windows\system32\grr\%version%\. It will also install a Windows Service, start it, and configure the registry keys to make it talk to the URL/server you specified during repack of the clients on the server.

  • For OSX you will see a pkg file, install the pkg. It will add a launchd item and start it.

  • For Linux you will see a deb and rpms, install the appropriate one. For testing purposes you can run the client on the same machine as the server if you like.

After install, hit Enter in the search box in the top left corner of the UI to see all of your clients that have enrolled with the server. If you don’t see clients, follow the troubleshooting steps.

Release notes

Each release of GRR brings some significant changes as the project’s code is moving quickly. This page tries to identify the key changes and issues to watch for when upgrading from one version to another.

Note that since the 3.2.0.1 release, client and server releases are no longer done separately (server debs include client templates).

Remember to back up your data-store and configs before upgrading GRR to a new version.

Server

3.4.3.1 (May 19 2021)

Regular release for Q2 2021.

  • Prometheus/Grafana support.

  • New flow: “Collect large file” that uploads a given path to Google Cloud Storage using a given signed URL.

  • New output plugin: Elasticsearch (thanks micrictor@ for the contribution!)

  • Fleetspeak-enabled setups now use streaming connections by default (agents keep connections open, thus reducing latency from ~5 minutes to under 1 minute).

  • API changes:

    • Introduced KillFleetspeak, RestartFleetspeakGrrService, DeleteFleetspeakPendingMessages, GetFleetspeakPendingMessages, GetFleetspeakPendingMessageCount API methods to provide Fleetspeak-specific capabilities for Fleetspeak-enabled clients.

    • Introduced ListParsedFlowResults and ListFlowApplicableParsers API methods for on-demand artifacts parsing.

  • Bugfixes and various minor enhancements:

    • Osquery doesn’t report results if the client returns an empty dataset.

    • Additional options can be specified when downloading timelines.

    • Introduced Hunt.default_client_rate configuration option.

    • Added Cron.interrogate_client_rate config option (defaults to 50).

  • Ongoing work on the UIv2: very early version of the next-gen UI can be checked via the <admin ui address>/v2 URL.

Note: this is the last release where Fleetspeak is considered an experimental feature. Next release is going to use Fleetspeak as a default option (while still keeping the legacy communication protocol as a deprecated option).

3.4.2.4 (October 14 2020)

Minor bug-fix release on top of v3.4.2.3.

  • grr_config_updater: Fixed a bug where the fleetspeak MySQL password setting was not applied correctly. Fleetspeak was effectively always using an empty MySQL password when connecting to the database.

3.4.2.3 (October 01 2020)

Quarterly release for Q3 2020.

  • GRR API:

    • Reference documentation is now available.

    • An OpenAPI spec is now available.

    • Support for starting file collection hunts involving multiple explicitly specified hosts is now available.

    • ArtifactCollectorFlowArgs, ArtifactFilesDownloaderFlowArgs:

      • use_tsk is now deprecated in favor of use_raw_filesystem_access

      • use_tsk will be kept for compatibility until 2021-04-01, existing users should migrate to using use_raw_filesystem_access

  • The NTFS virtual file system handler using libfsntfs is now the default for raw filesystem access on Windows. The TSK handler is being deprecated.

  • Experimental bundled fleetspeak (next generation communication framework) is now available on all platforms. This can be enabled using grr_config_updater.

  • Timeline collection flow now also collects the file creation timestamp on all platforms.

  • Flow and Hunt IDs now have a length of 16 hexadecimal digits.

  • A workaround for a bug in find_library affecting macOS 11 has been added.

  • Bugfixes and various enhancements.

3.4.2.0 (July 06 2020)

Quarterly release for Q2 2020.

  • New flows:

    • TimelineFlow for collecting a snapshot of an entire filesystem.

    • CollectEfiHashes and DumpEfiImage flows for macOS investigations.

  • New SplunkOutputPlugin to export Flow/Hunt results to Splunk.

  • New NTFS virtual file system handler for file system parsing using libfsntfs.

  • Support for custom Email addresses for GRR users. #555.

  • Support pre-filling the approval reason text field by appending ?reason=<pre-filled reason> to GRR URLs.

  • API Changes:

    • All GRR Protocol Buffers messages now have proper package declarations.

    • New method GetVersion.

    • API shell now validates GRR server version and fails if the API client is too old. (This can be disabled using the --no-check-version flag.)

  • Experimental bundled fleetspeak (next generation communication framework) on Debian server / clients. This can be enbled using grr_config_updater.

  • Bugfixes and various enhancements.

3.4.0.1 (December 18 2019)

This is a first fully Python 3-based release.

  • GRR server debian package now has to be installed on Ubuntu 18.

  • UpdateClient flow fixed for Ubuntu clients (see https://github.com/google/grr/issues/720).

  • A number of bugfixes and minor enhancements for YARA memory scanning.

3.3.0.8 (October 9 2019)

NOTE: Last Python 2-based release: further releases are expected to be Python 3-only.

Minor bug-fix release.

  • A number of REL_DB issues discovered since the last release fixed.

  • YARA scanning fixes and enhancements.

  • Using latest (in most cases) versions of dependencies (applies to both Python and JS).

3.3.0.4 (July 3 2019)

Minor bug-fix release.

  • Resolved an issue with approvals #719.

  • Resolved an issue with AdminUI failing when notification button was clicked, but no unread notifications were present.

3.3.0.3 (July 1 2019)

Minor bug-fix release.

  • Resolved an issue with client labels removal #716.

3.3.0.2 (June 28 2019)

Just a small bug-fix release.

  • Resolved stability issues in the MySQL backend #712.

  • Improved hostname handling #713.

  • Resolved BigQueryOutputPlugin failures #714.

  • Fixed issues with escaping path backslashes #715.

3.3.0.0 (May 22 2019)

  • [IMPORTANT] This is the first GRR release that works with the new relational data model. The legacy, AFF4-based data store is still available in this release but new GRR setups are encouraged to switch to the relational data store. Please see the documentation for details.

  • Rekall memory forensics support has been dropped from GRR. GRR still offers process memory acquisition and process memory scanning using the Yara library.

  • This release contains basic osquery integration for clients that have osquery preinstalled.

  • Custom monitoring code was removed in favor of prometheus.io integration.

  • Tons of small (and not-so-small) bug fixes and code health improvements.

  • API updates

    • ListFlowOutputPluginsLogs, ListFlowOutputPluginErrors, ListHuntOutputPluginLogs and ListHuntOutputPluginErrors API calls now always report batch_index and batch_size as 0 and no longer include PluginDescriptor into the reply.

    • ListHuntCrashes method no longer accepts the “filter” argument.

    • ListHunts no longer fills the “total_count” attribute of ApiListHuntsResult.

    • The ApiHunt protobuf no longer has an expires field. Instead, a duration field has been added which can be used to calculate the expiration date as start_time + duration. Note that if the hunt hasn’t been started, it does not have a start_time and, in consequence, it does not have an expiry time either.

    • The ApiModifyHuntArgs protobuf no longer has an expires field. Instead, a duration field has been added.

    • The artifact field of ApiUploadArtifactArgs no longer accepts an arbitrary byte stream but only proper strings. Since this field is ought to be the artifact description in the YAML format and YAML is required to be UTF-8 encoded, it makes no sense to accept non-unicode objects.

3.2.4.6 (Dec 20 2018)

This is just a small off-schedule release with some bugfixes.

  • Fixed a unicode bug that prevented Windows clients with non-latin usernames from checking in.

  • Fixed a bug in the grr_config_updater generate_keys command.

3.2.4.5 (Dec 17 2018)

  • [IMPORTANT] This release is the last GRR release to work on a legacy AFF4-based datastore. Next generation datastore will also work on top of MySQL but will have a completely different schema, meaning that you’ll lose historical data with the next GRR upgrade (unfortunately we can’t provide data migration tools).

  • [IMPORTANT] This release is the last GRR release to still include deprecated Rekall support. It will be removed completely with the next GRR release.

  • Dropped support for SQLite datastores. Current users with SQLite deployments will have to migrate to MySQL when upgrading. Please note that GRR does not offer any functionality for migrating data from SQLite databases to MySQL. The setup process for MySQL datastores has been revamped, and is easier and more intuitive now. Users will have the option of aborting installation if MySQL is not installed on a target system, and GRR will now proactively and transparently attempt to connect to a MySQL instance during initialization.

  • Improvements to the Python package structure.

  • A lot of Python 3-compatibility improvements.

  • Implementation of client-side artifact collector that tremendously improves latency of artifact collection.

  • Tons of small (and not-so-small) bug fixes and code health improvements.

API updates

  • Renamed the task_eta field of the ApiClientActionRequest object to leased_until.

  • Got rid of ListCronJobFlows and GetCronJobFlow in favor of ListCronJobRuns and GetCronJobRun. ListCronJobRuns/GetCronJobRun return ApiCronJobRun protos instead of ApiFlow returned by deleted ListCronJobFlows/GetCronJobFlow.

  • Changed CreateCronJob to accept newly introduced ApiCreateCronJobArgs instead of an ApiCronJob. ApiCreateCronJobArgs only allows to create hunt-based cron jobs.

  • All ApiFlowRequest responses do not fill the AFF4 specific request_state.request field anymore. Similarly, the task_id and payload fields in ApiFlowRequest.responses objects is not populated anymore starting from this release.

  • Flow log results returned by ApiListFlowLogsHandler do not contain the name of the flow the logs are for anymore.

  • The ListPendingGlobalNotifications and DeletePendingGlobalNotification API methods have been deleted, since GRR no longer supports global notifications. The corresponding protos ApiListPendingGlobalNotificationsResult and ApiDeletePendingGlobalNotificationArgs have been deprecated.

3.2.3.2 (Jun 28 2018)

  • This is an off-schedule release that fixes a client-repacking bug introduced in v3.2.3.0.

3.2.3.0 (Jun 25 2018)

  • We are planning to deprecate the SQLite datastore in favor of MySQL. Support for SQLite will likely get dropped in the coming few releases.

  • Significant progress has been made towards reimplementing GRR’s database model to use a relational schema. We expect to launch the new model - replacing the old one - in the next few months. The new model will not have a SQLite implementation (which is why we are dropping SQLite).

  • UI improvements and bugfixes.

  • Previously, when building and repacking client templates, config options in the ‘ClientBuilder’ context would leak into client config files. This will no longer happen, starting from this release. Config files that define client options inside a ‘ClientBuilder’ context might need fixing.

  • Api Updates:

    • GetGrrBinary API method now returns ApiGrrBinary object instead of a binary stream. The old behavior is preserved in a newly introduced GetGrrBinaryBlob method.

    • Additional fields were added to Api{Hunt,CronJob,Client}Approval protos: requestor and email_message_id.

    • ApiNotification.Type enum has been changed: 0 now means ‘UNSET’ (previously it meant ‘DISCOVERY’, but ‘DISCOVERY’ type is now deprecated).

    • ApiNotificationDiscoveryReference has been renamed to ApiNotificationClientReference.

    • ApiNotificationFileDownloadReference has been deprecated.

    • In ApiNotificationCronReference and ApiNotificationCronJobApprovalReference protos, the cron_job_urn field has been replaced with cron_job_id.

    • In the ApiNotificationHuntReference proto, the hunt_urn field has been replaced with hunt_id.

3.2.2.0 (Mar 12 2018)

  • As of this release, GRR’s legacy asciidoc documentation (previously hosted on github) has been deleted. https://grr-doc.readthedocs.io is now GRR’s only documentation repository.

  • GRR’s protobufs have been refactored into their own pip package grr-response-proto, which is now a dependency of grr-response-core.

  • User objects now get automatically created when using RemoteUserWebAuthManager (see Running GRR behind Nginx). Users who log in for the first time no longer need to do anything special to make notifications work.

  • grr-response-client’s pip dependency for Rekall has been updated to version 1.7.2rc1.

  • Functionality has been added to allow showing predefined warnings in the Admin UI when accessing clients with certain labels (AdminUI.client_warnings config option).

  • Scanning and dumping process memory with Yara has been made more resource-efficient, and now puts less strain on client machines.

  • A number of changes have been made to the GRR Api, some of which are not backwards compatible with Api client code expecting pre-3.2.2.0 responses and behavior:

    • hunt_pb2.ApiListHuntLogsResult now uses hunt_pb2.ApiHuntLog, a new proto, instead of jobs_pb2.Flowlog.

    • hunt_pb2.ApiListHuntErrorsResult now uses hunt_pb2.ApiHuntError, a new proto, instead of jobs_pb2.HuntError.

    • flow_pb2.ApiListFlowLogsResult now uses flow_pb2.ApiFlowLog, a new proto, instead of jobs_pb2.Flowlog.

    • The default age attribute has been removed from Api responses.

    • cron_pb2.ApiCronJob protos now have a cron_job_id field.

    • GetClientVersions Api call (/api/clients/<client_id>/versions) no longer includes metadata (last ping, last clock, last boot time, last crash time).

3.2.1.1 (Dec 6 2017)

  • The HTTPDatastore has been removed from GRR.

  • GRR now supports MySQL out of the box (when installed from the server deb).

  • GRR’s documentation has been revised and moved to https://grr-doc.readthedocs.io. The old asciidoc documentation (currently hosted on github) will be deleted before the next release.

  • SSO support has been added to the Admin UI, with the RemoteUserWebAuthManager.

  • Auto-refresh functionality for flows and hunts has been added to the GRR UI.

  • Support for per-client-average limits for hunts has been added.

  • GRR now uses the pytest framework to run tests. grr_run_tests is deprecated, and will be removed before the next release.

  • Support for scanning and dumping process memory using Yara has been added.

  • The SearchFileContent flow has been deprecated, in favor of equivalent FileFinder functionality.

  • zip/tar.gz archives downloaded from GRR will no longer use symlinks to avoid duplication of files - this makes them easier to work with.

3.2.0.1 (Sep 5 2017)

Starting with 3.2.0.1, we plan on releasing more frequently, since we have automated a large part of the release process. The recommended way of installing this version of GRR is with the server debian package, which includes client templates for all supported platforms. PIP packages, and a Docker image for the release will be made available at a later date.

  • IMPORTANT Before upgrading, make sure to back up your data store. This release has changes that may be incompatible with older versions of the data store. Also make copies of your configs - for reference, and in case you want to roll back the upgrade.

  • WARN Rekall is disabled by default. ‘Rekall.enabled: True’ line should be added to GRR configuration file in order for Rekall to work. There are known stability issues in Rekall and winpmem driver. Disabling it by default makes its ‘in dev/experimental’ status more explicit.

  • WARN Components are deprecated. The GRR client is now monolithic, there’s no need to upload or update components anymore. NOTE: The new GRR server will only be able to run Rekall-based flows (AnalyzeClientMemory, MemoryCollector) with the latest GRR client, since it expects the client not to use components.

  • WARN Rekall flows have been moved to ‘DEBUG’ category. They won’t be visible unless users enable ‘Mode: DEBUG’ in the settings menu (accessible via the settings button in top right corner of GRR AdminUI).

  • WARN A number of backwards-incompatible datastore schema changes were made. The nature of these changes is such that easy data migration between releases is not possible. Data loss of flows and hunts is possible. Please back up your current system if this data is critical.

  • WARN A number of deprecated configuration options were removed. If GRR fails to start after the upgrade, please check the server logs for ‘Unknown configuration option’ messages.

  • WARN Global Flows are no longer supported.

  • INFO Bugfixes, additionals tests and code health work in all parts of the system.

  • INFO HTTP API now covers 100% of GRR functionality. Python API client library significantly extended. API call routers implemented, allowing for flexible authorization.

  • INFO ‘Download as’ functionality added to flows and hunts. Supported formats are CSV, YAML and SQLite.

  • INFO Build process for clients and server is significantly improved. Client templates for all supported platforms, as well as a release-ready server deb, get built on every github commit.

0.3.0-7 to 3.1.0.2

Note that we shifted our version string left to reduce confusion, see the documentation on GRR versions.

  • WARN Quite a few config options have changed. We recommend moving any customizations you have into your server.local.yaml, and using the main config file supplied with the new version. Some important specific ones are called out here.

  • WARN Config option Client.control_urls has been replaced with Client.server_urls. You will need to make sure your polling URL is set in the new variable.

  • WARN We have completely rewritten how memory collection and analysis works inside GRR. Rekall is now responsible for all driver loading, memory collection, and memory analysis. It also replaces the functionality of the Memory Collector flow. This means you must upgrade your clients to be able to do memory collection and analysis.

  • WARN Repacking of old client templates no longer works due to changes necessary for repacking different versions. You will need to use the new client templates that include a build.yaml file.

  • WARN Config option Cron.enabled_system_jobs (a allowlist) was replaced with Cron.disabled_system_jobs (a blocklist) to make adding new cronjobs easier. You will need to remove Cron.enabled_system_jobs and if you made customizations here, translate any jobs you want to remain disabled to the new list.

  • WARN We changed the format for the Knowledge Base protocol buffer to remove a duplication of the users component which was storing users information twice and complicating code. This means that all clients need a fresh interrogate run to populate the machine information. Until the machines run interrogate the %%users.homedir%% and similar expansions won’t work.

  • WARN The compiler we use to build the Mac client template has become more aggressive with optimizations and now emits code optimized to specific processor versions. The mac client will not run on a processor older than our build machine - an early 2011 Macbook with the Intel Sandy Bridge architecture. If you need to run GRR on, for example, Core 2 duo machines, you will need a custom client built on one of those machines.

  • WARN We no longer support running the client on Windows XP, it’s difficult to make it work and we have no use for an XP client ourselves. See here for our OS support statement.

  • INFO Strict context checking was added for config files in this commit, which exposed a number of minor config typo bugs. If you get InvalidContextError on upgrade, you need to update your config to fix up the typos. The config/contexts.py file is the canonical reference.

Upgrade procedure:

  1. Make backups of important data such as your configs and your database.

  2. Upgrade to xenial

  3. Clear out any old installations: sudo rm -rf /usr/local/bin/grr_*; sudo rm -rf /usr/local/lib/python2.7/dist-packages/grr/

  4. Install deb package

  5. Backup then remove your /etc/grr/grr-server.yaml. Make sure any customizations you made are stored in /etc/grr/server.local.yaml.

  6. Run grr_config_updater initialize

0.3.0-6 to 0.3.0-7

  • INFO We moved to using the open source binplist, rather than bundling it with GRR. If you are getting “ImportError: cannot import name binplist”, make sure you have installed the binplist package and deleted grr/parsers/binplist* to clean up any .pyc files that might have been created.

  • INFO If you have troubles with the rekall efilter library failing to import, try deleting all of your rekall-related .pyc’s and make sure you’re installing the version in requirements.txt. See this bug for full details.

0.3.0-5 to 0.3.0-6

This version bump was to keep in step with the client, which got a new version because of a memory collection bug.

  • WARN The artifact format changed in a way that is not backwards compatible. You’ll need to delete any artifacts you have in the artifact manager, either through the GUI or on the console with aff4.FACTORY.Delete("aff4:/artifact_store") before you upgrade. If you forget you should be able to use the console or you can roll back to this commit before the artifact change.

  • WARN Keyword search is now based on an index that is built at weekly interrogate time, which means decomissioned clients won’t appear in search results. To populate all of your historical clients into the search index, use the code snippet on this commit.

0.3.0 to 0.3.0-5

  • WARN Rekall made some fundamental changes that mean rekall in the 0.3.0-5 server won’t work with rekall in 3.0.0.3 clients, so a client upgrade to 3.0.0.5 is required. Non-rekall GRR components should continue to work.

  • WARN The enroller component was removed, and is now handled by the worker. You will need to stop those jobs and delete the relevant upstart/init scripts.

  • INFO We now check that all config options are defined in the code and the server won’t start if they aren’t. When you upgrade it’s likely there is some old config that will need cleaning up. See the graveyard below for advice.

0.2.9 to 0.3.0

  • WARN After install of new deb you will need to restart the service manually.

  • INFO To get the new clients you will want to repack the new version with sudo grr_config_updater repack_clients. This should give you new Windows, Linux and OSX clients.

  • INFO Client themselves will not automatically be upgraded but should continue to work.

0.2-8 to 0.2-9

  • WARN After install of new deb you will need to restart the service manually.

  • WARN Users have changed from being managed in the config file to being managed in the datastore. This means your users will not work. We haven’t migrated them automatically, please re-add with sudo grr_config_updater add_user <user>

  • INFO To get the new clients you will want to repack the new version with sudo grr_config_updater repack_clients. This should give you new Windows, Linux and OSX clients.

  • INFO Client themselves will not automatically be upgraded but should continue to work.

Client

3.2.0.1 -

Starting from 3.2.0.1, we the same version string for the client and server, since they get released at the same time.

3.0.7.1 to 3.1.0.0

Note that we skipped some numbers to make versioning simpler and reduce confusion, see the documentation on GRR versions.

  • WARN We changed rekall to be a independently updatable component in the client, which is a backwards incompatible change. You must upgrade your clients to 3.1.0.0 if you want to use memory capabilities in the 3.1.0 server.

  • WARN Our previous debian package added the GRR service using both upstart and init.d runlevels. This caused some confusion on systems with ubuntu upstart systems. We detect and remove this problem automatically with the new version, but since it is a config file change you need to specify whether to install the new config or keep the old one, or you will get a config change prompt. New is preferred. Something like sudo apt-get -o Dpkg::Options::="--force-confnew" grr.

3.0.0.7 to 3.0.7.1

  • INFO 3.0.7.1 is the first version of GRR that will work on OS X El Capitan. The new OS X System Integrity Protection meant we had to shift our install location from /usr/lib/ to /usr/local/lib.

  • INFO We changed our version numbering scheme for the client at this point to give us the ability to indicate multiple client versions that work with a server version. So 3.0.7.* clients will all work with server 0.3.0-7.

3.0.0.6 to 3.0.0.7

  • WARN Linux and OS X clients prior to 3.0.0.7 were using /etc/grr/client.local.yaml as the local writeback location. For 3.0.0.7 this was changed to /etc/%(Client.name).local.yaml where the default is /etc/grr.local.yaml. If you wish to preserve the same client IDs you need to use platform management tools to copy the old config into the new location for all clients before you upgrade. If you don’t do this the clients will just re-enrol and get new client IDs automatically.

3.0.0.5 to 3.0.0.6

  • INFO 3.0.0.5 had a bug that broke memory collection, fixed in this commit. We also wrote a temporary server-side workaround, so upgrading isn’t mandatory. 3.0.0.5 clients should still work fine.

3.0.0.3 to 3.0.0.5

(We skipped a version number, there’s no 3.0.0.4)

3.0.0.2 to 3.0.0.3

  • WARN A change to OpenSSL required us to sign our CSRs generated during the enrollment process. This wasn’t necessary previously and provided no benefit for GRR so we had gained some speed by not doing it. Since new OpenSSL required it, we started signing the CSRs, but it meant that the 3.0.0.3 server will reject any 3.0.0.2 clients that haven’t already enrolled (i.e. they will see a HTTP 406). Old 3.0.0.2 clients that have already enrolled and new 3.0.0.3 clients will work fine. This basically just means that you need to push out new clients at the same time as you upgrade the server.

Config Variable Graveyard

Sometimes config variables get renamed, sometimes removed. When this happens we’ll try to record it here, so users know if local settings should be migrated/ignored etc.

You can verify your config with this (root is required to read the writeback config)

sudo PYTHONPATH=. python ./run_tests.py --test=BuildConfigTests.testAllConfigs

  • AdminUI.team_name: replaced by Email.signature

  • ClientBuilder.build_src_dir: unused, effectively duplicated ClientBuilder.source

  • ClientBuilder.executables_path: ClientBuilder.executables_dir

  • Client.config: unused. Now built from Client.config_hive and Client.config_key

  • Client.config_file_name: unused

  • Client.location: replaced by Client.control_urls

  • Client.package_maker_organization: replaced by ClientBuilder.package_maker_organization

  • Client.tempdir: replaced by Client.grr_tempdir and Client.tempdir_roots

  • Email.default_domain: essentially duplicated Logging.domain, use that instead.

  • Frontend.processes: unused

  • Nanny.nanny_binary: replaced by Nanny.binary

  • NannyWindows.* : replaced by Nanny.

  • PyInstaller.build_root_dir: unused, effectively duplicated ClientBuilder.build_root_dir.

  • Users.authentication: unused, user auth is now based on aff4:/users objects. Use config_updater to modify them.

  • Worker.task_limit: unused

  • Worker.worker_process_count: unused

  • Cron.enabled_system_jobs (a allowlist) was replaced with Cron.disabled_system_jobs (a blocklist). Cron.enabled_system_jobs should be removed. Any custom jobs you want to stay disabled should be added to Cron.enabled_system_jobs.

  • Client.control_urls: renamed to Client.server_urls.

Frequently Asked Questions

Who wrote GRR and Why?

GRR started at Google as a 20% project and gained in popularity until it became fully-supported and open sourced. The primary motivation was that we felt the state of the art for incident response was going in the wrong direction, and wasn’t going to meet our cross platform, scalability, obfuscation or flexibility goals for an incident response agent.

Additionally, we believe that for things to progress in security, everyone has to up their game and improve their capabilities. We hope that by open sourcing GRR, we can foster development of new ways of doing things and thinking about the problem, get it into the hands of real organizations at reasonable cost, and generally push the state of the art forward.

We are getting significant code contributions from outside of Google, and have close relationships with a number companies running large-scale deployments.

Why is the project called GRR?

When using other tools, we found ourselves making the sound “grrr” a lot, and it just kind of stuck. GRR is a recursive acronym, in the tradition of GNU and it stands for GRR Rapid Response. Not GRR Response Rig or Google Rapid Response which it is sometimes mistaken for.

Is GRR production ready?

As of Aug 2015 GRR is running at large scale both inside and outside of Google. The largest opensource deployment we know of is roughly 30k machines, there’s another company at around 15k, and quite a few around 2k. Google’s deployment is bigger than all of those put together, although there are some differences in the codebase (see below).

Should I expect to be able to install and just start running GRR?

Yes, for basic use cases.

But if you want to do a large-scale enterprise deployment, it is probably best to think about GRR as a 80% written software project that you could invest in. The question then becomes: instead of investing X million in product Y to buy something, should I instead invest 25% of that in GRR and hire a dev to contribute, build and deploy it? On the one hand that gives you control and in-house support, on the other, it is a real investment of resources.

If you are selling GRR internally (or to yourself) as a free <insert commercial IR product here>, your expectations will be wrong, and you may get disillusioned.

Can the GRR team provide me with assistance in getting it setup?

The core GRR team cares about the open source project, but in the end, our main goals are to build something that works for us. We don’t, and won’t offer a helpdesk, professionally curated documentation, nor someone you can pay money to help you out if something goes wrong. We aren’t providing feeds or consulting services, and have nothing direct to gain from offering the platform. If you need something pre-packaged and polished, GRR probably isn’t right for you (at least in the short-medium term). For a large deployment you should expect to fix minor bugs, improve or write documentation, and actively engage with the team to make it successful.

If someone is willing to code, and has invested some time learning we will do what we can to support them. We’re happy to spend time on VC or in person helping people get up to speed or running hackathons. However, the time that the developers invest in packaging, building, testing and debugging for open source is mostly our own personal time. So please be reasonable in what and how you ask for assistance. We’re more likely to help if you’ve contributed documentation or code, or even filed good bug reports.

I’m interested in GRR but I, or my team need some more convincing. Can you help?

The core GRR team has invested a lot in the project, we think its pretty awesome, so the team happy to talk, do phone calls, or chat with other teams that are considering GRR. We’ve even been known to send house-trained GRR engineers to companies to talk with interested teams. Just contact us directly. You also can corner one of us, or at least someone from the team, or someone who works on GRR at most leading forensics/IR type conferences around the world.

I’ve heard that there are secret internal versions of GRR that aren’t open sourced that may have additional capabilities. Is that true?

GRR was always designed to be open sourced, but with any sufficiently complex “enterprise” product you expect to integrate it with other systems and potentially even with proprietary technology. So its true that some of the core developers time is spent working on internal features that won’t be released publicly. The goal is to ensure that everything useful is released, but there are some things that don’t make sense. Below are listed some of the key differences that may matter to you:

  • Datastore/Storage: At Google we run GRR on a Bigtable datastore (see below for more detail), but we have abstracted things such that using a different datastore is very simple. The MySQLAdvanced datastore is available to open source are actively used at real scale outside of Google.

  • Security and privacy: The open source version has minimal controls immediately available for user authentication, multi-party authorization, privacy controls, logging, auditing etc. This is because these things are important enough for them to be custom and integrated with internal infrastructure in a large deployment. We open source the bits that make sense, and provide sensible hooks for others to use, but full implementations of these may require some integration work.

  • Machine handling and monitoring: Much of the infrastructure for running and monitoring a scalable service is often built into the platform itself. As such GRR hasn’t invested a lot in built-in service or performance monitoring. We expose a lot of statistics, but only display a small subset in the UI. We expect companies to have package deployment (SCCM/munki/casper/apt/yum etc.), config management (chef/puppet/salt etc.), and server monitoring (nagios/cacti/munin/zabbix/spiceworks etc.).

Differences will be whittled away over time as the core GRR team runs open source GRR deployments themselves. That means you can expect most of these things to become much less of an issue over time.

Why was support for SQLite dropped?

Our intent for SQLite was for it to be used as a casual, easy-setup datastore for testing/evaluating GRR. It doesn’t scale well, and is not recommended for actual production use. When developing GRR’s new relational database model, we decided that continuing to have support for SQLite was not worth the engineering effort it would take to build a SQLite implementation of the new model.

I heard GRR was designed for Bigtable and now Google has a Cloud Bigtable service. Can I use it?

Internally we use Bigtable, but the internal API is very different so the code cannot be used directly. The Cloud Bigtable service uses an extension of the HBase API. We’d like to write a GRR database connector that can use this service, but (as at Aug 2015) the Bigtable service is still in Beta and the python libraries to interact with it are still being developed, so it isn’t currently a high priority.

What operating system versions does the client support?

We try to support a wide range of operating systems because we know it’s often the old forgotten machines that get owned and need GRR the most. Having said that ‘support’ is minimal for very old operating systems, we’re a small team and we don’t have the time to test the client on every possible combination. So the approach is basically to try and keep support for old systems by building compiled dependencies in a way that should work on old systems.

Windows

  • Well tested on: 64bit Windows 7+ workstations, 64bit Win2k8+ servers.

  • Should probably work: 32bit versions of the above, Windows Vista+ workstations. (We’ve stopped building and releasing 32bit versions since release 3.4.0. It should be possible to build a client from source though.)

OS X

  • Well tested on: 64bit 10.8 (Mountain Lion)+ Note that 10.11 (El cap)+ systems require a 3.0.7.1+ client due to the OS X System Integrity Protection changes. And sleuthkit hasn’t caught up to filesystem changes from Yosemite onwards. The compiler we use to build the Mac client template has become more aggressive with optimizations and now emits code optimized to specific processor versions. The mac client will not run on a processor older than our build machine - an early 2011 Macbook with the Intel Sandy Bridge architecture. If you need to run GRR on, for example, Core 2 duo machines, you will need a custom client built on one of those machines.

Linux

  • Well tested on: 64bit Ubuntu Lucid+, CentOS 5.11+

  • Should probably work: 32bit versions of the above, and essentially any system of similar vintage that can install a deb or rpm.

What operating system versions does the server support?

We only support 64bit Ubuntu 18.04.

We don’t provide a 32-bit server version since standing up new 32-bit linux servers is not something rational people do, and there are many places you can get 64-bit virtual servers for ~free. We use the “amd64” notation, but this just means 64-bit, it runs fine on Intel.

What hardware do I need to run GRR?

This is actually a pretty tough question to answer. It depends on the database you choose, the number of clients you have, and how intensively you hunt. Someone who wants to do big collection hunts (such as sucking all the executables out of System32) will need more grunt and storage than someone who mainly wants to check for specific IOCs and investigate single machines.

But to give you some data points we asked some of the GRR users with large production installs about the hardware they are using (as at October 2015) and provide it here below:

32k clients:

  • Workers: AWS m4.large running 3 worker processes

  • HTTP frontends: 6-10 x AWS m4.large. Sits behind an AWS Elastic Load Balancer.

  • Datastore (SQLite): 5 x AWS m4.2xlarge. m4.2xlarge is used when running intensive enterprise hunts. During normal usage, m4.large is fine.

  • AdminUI: 1 m3.large

15k clients:

  • Workers and HTTP frontends: 10 x 4 core 8GB RAM virtual servers running 1 worker + 1 frontend each

  • Datastore (MySQLAdvanced): 16 core 256G ram 8x10k drives. 128G RAM was sufficient, but we had the opportunity to stuff extra RAM in so we did.

  • AdminUI: 12 core 24G RAM is where we left the UI since it was spare hardware and we had a lot of active users and the extra RAM was nice for faster downloads of large datasets. It was definitely overkill and the backup was on a 4 core 8GB of RAM VM and worked fine during maintenance stuff.

7k clients:

Run in AWS with c3.large instances in two autoscaling groups.

  • Workers: 4 worker processes per server. The weekly interrogate flow tends to scale up the servers to about 10 systems, or 40 workers, and then back down in a couple of hours.

  • HTTP frontends and AdminUI: Each server has apache running a reverse proxy for the GRR AdminUI. At idle it uses just a t2.small, but whenever there is any tasking it scales up to 1-3 c3.large instances. Sits behind an AWS Elastic Load Balancer.

  • Datastore (MySQLAdvanced): AWS r3.4xlarge RDS server. RDS instance is optimized for 2000 IOPS and we’ve provisioned 3000.

How do I handle multi-organisational deployments?

Bake labels into clients at build time, and use a “Clients With Label” hunt rule to hunt specific groups of clients separately.

Which cloud should I deploy in? GCE? EC2? Azure?

Google Compute Engine (GCE) of course :) We’re working on making cloud deployment easier by dockerizing and building a click-to-deploy for GCE. Our focus will be primarily on making this work on GCE, but moving to a docker deployment with orchestration will simplify deployment on all clouds. The largest cloud deployments of GRR are currently on EC2, and we hope the community will be able to share configuration and HOWTOs for this and other cloud deployments.

Where/how do you do your data analysis?

We mostly do this outside of GRR using an internal system very similar to BigQuery. GRR supports Output Plugins that send hunt results to other systems (e.g. BigQuery, Splunk) with minimal delay.

When will feature X be ready?

Generally our roadmap on the main project page matches what we are working on, but we reserve the right to miss those goals, work on something entirely different, or sit around a fire singing kumbaya. Of course, given this is open source, you can add the feature yourself if it matters.

Who is working on GRR?

GRR has around 5 full-time software engineers working on it as their day job, plus additional part time code contributors. The project has long term commitment.

Why aren’t you developing directly on open source?

Given we previously had limited code contribution from outside, it was hard to justify the extra effort of jumping out of our internal code review and submission processes. That has now changed, we are syncing far more regularly (often multiple times per week), and we are working on code structure changes that will make it easier for us to develop externally.

Why is GRR so complicated?

GRR is complicated. We are talking about a distributed, asynchronous, cross platform, large scale system with a lot of moving parts. Building that is a hard and complicated engineering problem. This is not your average pet python project.

Having said that, the most common action of just collecting something from machines and parsing what you get back has been made significantly easier with the artifacts system. This allows you to specify complex multi-operating system collection tasks with just a few lines of YAML, and collect any of the hundreds of pre-defined forensic artifacts with the click of a button.

What are the commercial competitors to GRR?

Some people have compared GRR functionality to Mandiant’s MIR, Encase Enterprise, or F-Response. There is some crossover in functionality with those products, but we don’t consider GRR to be a direct competitor. GRR is unlikely to ever be the product for everyone, as most organizations need consultants, support and the whole package that goes with that.

In many ways we have a way to go to match the capabilities and ease of use of some of the commercial products, but we hope we can learn something off each other, we can all get better, and together we can all genuinely improve the security of the ecosystem we all exist in. We’re happy to see others use GRR in their commercial consulting practices.

Where is the logout button?

There isn’t one. We ship with basic auth which doesn’t really handle logout, you need to close the browser. This is OK for testing, but for production we expect you to sit a reverse proxy in front of the UI that handles auth, or write a webauth module for GRR. See the Authentication to the AdminUI section for more details.

How do I change the timezone from UTC?

You can’t. Technically it’s possible, but it’s a lot of work and we don’t see much benefit. You should run your GRR server in the UTC timezone.

In the datastore GRR stores all timestamps as microseconds from epoch (UTC). To implement timezone customization we could store a user’s local preference for timezone and daylight savings based on a location, and convert all the timestamps in the UI, but there is a lot of complexity involved. We’d need to make sure each user gets the correct time and timezone displayed in all parts of the UI, handle the shifting winds of daylight savings correctly, provide clear input options on forms that accept timestamps, and make sure all the conversions were correct. All of that adds up to lots of potential to display or store an incorrect time, which is not what you want from a forensics tool.

It’s common to have GRR users split across different timezones and a client fleet in many timezones. If we went to all of the effort above, all we have really achieved is fragmentation of each user’s view of the system. Each user is still going to have to analyse and reason about events on clients in a different timezone to their own. But now we have made collaborative analysis harder: instead of being able to simply copy a timestamp out of the UI into a ticket or handoff timeline analysis notes to another shift in another timezone they will have to convert timestamps back into UTC manually for consistency.

For forensic analysis and timeline building, UTC always makes the most sense. We have added a UTC clock to the top right hand corner of the UI to help reason about the timestamps you see.

Is there any relation between the grr pip package and Google’s GRR?

No. The naming collision is just a coincidence, and you might actually run into issues if you install the two in the same Python environment.

Publications

Papers

A scalable file based data store for forensic analysis, by Flavio Cruz, Andreas Moser, Michael Cohen. DFRWS 2015 Europe.

Hunting in the enterprise: Forensic triage and incident response, by Moser, Andreas, and Michael I. Cohen. Digital Investigation, 2013.

Distributed forensics and incident response in the enterprise, by M.I. Cohen, D. Bilby, G. Caronni. Digital Investigation, 2011.

Presentations

GRR Meetup: GRR Users Meetup: Fleetspeak, API client, Go rewrite @ Mar 2017 GRR Meetup by A. Moser, M. Bushkov, B. Galehouse, M. Lakomy. Video.

GRR Meetup: 3.1.0 Release @ Apr 2016 GRR Meetup by G. Castle and M. Cohen. Video.

GRR Meetup: API Edition @ Nov 2015 GRR Meetup by G. Castle and M. Bushkov. Covers API basics and discusses where we are headed with API dev.

GRR Hunting Best Practices @ Oct 2015 GRR Meetup by G. Castle.

Tactical use of GRR Artifacts, Fuse, Plaso and Timesketch by D. Bilby Demo Video (youtube).

“Intro to GRR” for the Open Source Network Security Monitoring Group @ UIUC Feb 2015. Video by G. Castle

“GRR: Find all the badness, collect all the things” @ Blackhat 2014 Video by G. Castle. Also presented a 20 min version of the same talk for Hacker Hotshots.

OSDFC 2012 GRR Overview, by D. Bilby

Workshops

GRR Workshop at CERN, February 2018 slides by M. Bushkov, B. Galehouse, M. Lakomy, and A. Moser

ICDF2C Prague 2017 Workshop slides by M. Bushkov and B. Galehouse

DFRWS US 2015 Workshop slides and scripts to setup your own workshop infrastructure by G. Castle.

DFRWS EU 2015 Workshop workshop slides by A. Moser

OSDF 2013 workshop presentation by D. Bilby.

Podcasts

GRR Rapid Response on Down the Security Rabbithole Jun 2014 by G. Castle.

Whitepapers

GRR Forensic Artifacts White Paper, by G. Castle

External Presentations

These are presentations about GRR by people outside the core GRR team.

“Human Hunting” @BSidesSF 2015 by Sean Gillespie. Covers how Yahoo is using GRR. Video.

“GRR Rapid Response: Remote Live Forensics for Incident Response” @ Linux Security Summit Aug 2015 by Sean Gillespie.