Ranger Policy Deep Dive: policy JSON anatomy, resource matching, the evaluation algorithm worked by hand, policies as code and debugging denials

Anatomy of a policy

Every policy belongs to one service (an instance such as cm_hive or hdfs_prod) whose service definition fixes the resource hierarchy and the access types. A Hive access policy, as returned by the API, looks like this, trimmed to the fields that matter:

{
  "service": "cm_hive",
  "name": "sales_analysts_read",
  "policyType": 0,
  "policyPriority": 0,
  "isEnabled": true,
  "isAuditEnabled": true,
  "policyLabels": ["owner:data-platform", "ticket:DP-1412"],
  "resources": {
    "database": { "values": ["sales"], "isExcludes": false },
    "table":    { "values": ["orders", "returns"] },
    "column":   { "values": ["*"] }
  },
  "policyItems": [
    { "groups": ["analysts"], "accesses": [ { "type": "select", "isAllowed": true } ] }
  ],
  "allowExceptions": [
    { "users": ["contractor_bob"], "accesses": [ { "type": "select", "isAllowed": true } ] }
  ],
  "denyPolicyItems": [],
  "denyExceptions": [],
  "validitySchedules": [ { "startTime": "2026/10/01 00:00:00", "endTime": "2027/03/31 23:59:59", "timeZone": "UTC" } ]
}

Two details catch people. Exceptions apply only within the policy that declares them: an allow exception for contractor_bob here does not deny him if another policy allows him. And policyLabels are free-form, which makes them the natural place for ownership and change-ticket metadata that tooling can enforce.

How resources match

A policy applies to a request only if every level of its resource definition matches the requested resource. Values may contain the wildcards * and ?, and several values in one level are alternatives. Matching details vary by service, and three behaviours matter most.

• Recursion in path-based services. In HDFS, /data/sales with isRecursive: true covers everything below it; with recursion off it covers that exact path only. A common surprise is that a non-recursive policy on a directory allows listing it but not reading the files inside.
• Excludes. With isExcludes: true, a level matches everything except the listed values. database: ["hr"], isExcludes: true means every database but hr. It is compact and easy to misread, so label such policies clearly.
• Macros. Resource values can use the {USER} macro, so a single HDFS policy on /user/{USER} gives every user their own home directory without a policy per person.

For Hive, the resource levels are database, table and column (or UDF, or URL, depending on the resource type chosen). A query that reads three columns is checked for each column, and a query using SELECT * needs access to every column of the table. That is why column-level policies can break queries that seem to touch nothing sensitive.

Matching is also where tags enter. If Apache Atlas has classified a column as PII and the tag service is linked to the resource service, the plugin's tag enricher attaches that tag to the request, and tag-based policies written against PII apply alongside the resource policies.

The evaluation algorithm

Ranger's plugin evaluates a request in a fixed order. Tag-based policies come first: if a tag policy denies the access the request is denied, otherwise if a tag policy allows it the request is allowed, and only if no tag policy gives an answer (or the resource has no tags) do resource-based policies decide. Within resource policies, override-priority policies are evaluated before normal ones, so an override can allow what a normal policy denies. That ordering is documented for normal-priority policies; how an override resource policy interacts with a tag deny is not something to assume, so test that combination in staging before relying on it.

Within a priority level, the decision follows the deny-over-allow rule. As pseudocode:

def decide(request, policies):
    for priority in (OVERRIDE, NORMAL):
        matching = [p for p in policies
                    if p.enabled and p.priority == priority
                    and p.resources_match(request.resource)
                    and p.within_validity(request.time)]
        for p in matching:
            if p.item_matches(p.deny_items, request) and not p.item_matches(p.deny_exceptions, request):
                return DENY, p.id
        for p in matching:
            if p.item_matches(p.allow_items, request) and not p.item_matches(p.allow_exceptions, request):
                return ALLOW, p.id
    return NOT_DETERMINED, None   # HDFS may fall back to native permissions; most services deny

This is a model for reasoning, not the plugin's code, which caches evaluators, short-circuits and audits as it goes, but it predicts outcomes correctly for ordinary policies. item_matches is true when the request's user, one of its groups or one of its roles is listed, the access type is among the item's accesses, and every condition on the item, such as an IP range or an expression, evaluates true.

The not-determined case deserves attention. For HDFS, if the plugin is configured to fall back to Hadoop authorization, a request no Ranger policy decides is checked against the POSIX permissions and ACLs on the file, described in HDFS permissions and ACLs. That keeps legacy access working during migration, and it also means a missing Ranger policy is not a denial. A policy with the deny-all-else option set closes that gap for the resources it covers.

Worked example: one query, three policies

Take a request: user ana, groups analysts and emea, runs SELECT order_id, card_number FROM sales.orders. Atlas has tagged card_number as PCI. The relevant policies:

1. Tag policy P10 on PCI: deny select to group analysts, deny exception for group pci_auditors.
2. Resource policy P21 (normal): allow select on sales.orders.* to group analysts.
3. Resource policy P22 (normal): deny select on sales.*.* to group emea, deny exception for user ana.

Ranger evaluates each column. For order_id, which has no tags, resource policies decide. P22 matches and its deny item covers group emea, but ana is in its deny exception, so P22 does not deny. P21's allow item covers analysts, so order_id is allowed. For card_number, the PCI tag brings in P10; ana is in analysts and not in pci_auditors, so the tag policy denies and evaluation stops. The query fails with a permission error on card_number, and the audit record names P10.

Two lessons. First, removing ana from emea would not have changed anything; the tag deny is what blocked her. Second, if the business wants analysts to see masked card numbers rather than nothing, the right tool is a masking policy (policy type 1 with a mask type such as MASK_SHOW_LAST_4) rather than a deny, combined with removing the tag deny for that group.

Policies as code with the REST API

Clicking policies into the admin UI does not scale and leaves no review trail. Ranger's public REST API under /service/public/v2/api supports listing, fetching, creating, updating and deleting policies, which is enough to keep policies in Git and apply them from a pipeline.

# Fetch one policy by service and name
curl -s -u admin:*** "https://ranger.example.com:6182/service/public/v2/api/service/cm_hive/policy/sales_analysts_read"

# List all policies of a service
curl -s -u admin:*** "https://ranger.example.com:6182/service/public/v2/api/service/cm_hive/policy"

A sync script reads policy files from the repository and creates or updates them idempotently, keyed by service and name:

import json, pathlib, requests

BASE = 'https://ranger.example.com:6182/service/public/v2/api'
S = requests.Session(); S.auth = ('svc_ranger_sync', open('/run/secrets/ranger').read().strip())

def apply(policy):
    url = f"{BASE}/service/{policy['service']}/policy/{policy['name']}"
    r = S.get(url)
    if r.status_code == 404:
        S.post(f'{BASE}/policy', json=policy).raise_for_status()
        return 'created'
    r.raise_for_status()
    current = r.json()
    policy['id'] = current['id']
    if {k: current.get(k) for k in strip(policy)} == strip(policy):
        return 'unchanged'
    S.put(f"{BASE}/policy/{current['id']}", json=policy).raise_for_status()
    return 'updated'

def strip(p):   # compare only fields the Git file defines, minus server-managed ones
    return {k: v for k, v in p.items() if k not in ('id', 'guid', 'version', 'createTime', 'updateTime', 'createdBy', 'updatedBy')}

for f in sorted(pathlib.Path('policies').glob('*/*.json')):
    print(f, apply(json.loads(f.read_text())))

Run it with a dedicated service account that has admin rights only for the services the pipeline owns, and decide how to handle drift: either report policies that exist in Ranger but not in Git, or delete them. Reporting is safer to start with.

Testing policies before they ship

Because the evaluation rules are deterministic, policies can be tested. Three layers catch most mistakes.

• Static lint in CI. Reject policies that allow the public group on wildcard resources, that lack owner labels, that use excludes without a comment, or whose validity schedule has already ended.
• Decision tests. Keep a table of expected outcomes, such as (ana, select, sales.orders.card_number, DENY), and evaluate it against the policy files with a model like the pseudocode above. This catches changes that silently widen access.
• Staging replay. Apply the change to a staging Ranger with the same service definitions, run the test table as real users through beeline or the HDFS client, and compare the audit results with the expected outcomes.

The model will not cover every service-specific nuance, which is why the staging step exists, but it makes most review conversations concrete: the pull request shows which decisions changed.

Debugging a denial

When someone reports access denied, work from the audit record rather than from the policy list. Each Ranger audit event records the user, resource, access type, result, the policy ID that decided it and the enforcer. An enforcer of ranger-acl means a Ranger policy decided; for HDFS, hadoop-acl means no policy decided and native permissions were used. A policy ID of -1 means no policy matched.

If the audit shows no event at all, the request may never have reached the plugin, or the plugin may not be running. If the decision contradicts the policy you see in the UI, check whether the plugin has the latest policies: plugins poll the admin server at an interval (30 seconds by default for most) and cache policies on local disk, so a change can take a poll cycle to apply, and an unreachable admin server leaves the plugin on its last cached version. The admin UI shows each plugin's last download and activation times. Finally, check group membership as the plugin sees it: Ranger resolves groups from its user sync source or from the service, and a user added to an LDAP group yesterday may not be in that group in Ranger yet. Many group problems turn out to be Kerberos principal mapping problems, where the short name differs from the user Ranger knows.

Failure modes and trade-offs

Failure modes and trade-offs to plan for:

• Policy sprawl. Hundreds of overlapping per-team policies become impossible to reason about. Prefer roles and groups over users, and tags over per-table policies for data classes.
• Deny overuse. Denies are absolute within their priority and hard to override cleanly. Prefer structuring allows narrowly; keep denies for real prohibitions.
• Silent fallback. HDFS fallback to native ACLs hides missing policies. Turn it off once migration is complete.
• Expired schedules. A validity schedule that ends quietly removes access on a weekend; alert before expiry.

The core trade-off is between central control and local agility: a single policy set gives auditors one place to look, but every team now depends on the people who can change it. Policies as code with delegated review keeps the control while giving teams a path to change.

What to do next

Steps to take on a real Ranger deployment:

• Export all policies of each service through the REST API and commit them to a repository as a baseline.
• Add owner and ticket labels to every policy and lint for them in CI.
• Write a decision table for your ten most sensitive resources and test it on every change.
• Check which services fall back to native permissions and plan to disable the fallback.
• Move data-class rules such as PII and PCI to tag policies backed by Atlas classifications.
• Practise debugging one denial end to end from its audit record, including plugin sync time and group resolution.