Mastering Mixed Public & Private Files in Drupal 10 with S3fs (The "Private Bucket" Method)

Summary

Implementing a file system in Drupal 10 that securely handles sensitive documents while serving public images—all from a single, Private AWS S3 bucket—is notoriously difficult.

The challenge is compounded in multi-site environments (like Acquia) where standard public:// and private:// schemes might conflict. Furthermore, the S3FS module reserves specific folder names, crashing if you try to use them manually.

This article details the definitive solution: using the native s3:// scheme for isolation, routing URLs through a secure Base64 Proxy Controller, and using "safe" folder prefixes (open/ and file-private/) to bypass S3FS conflicts.

Image

The Core Problem: The "View Media" Paradox

Before writing code, it is critical to understand why standard Drupal permissions fail for this requirement.

Drupal controls file access based on the Parent Entity (the Media or Node). It does not natively control access based on the Field.

1. The Requirement: You want Anonymous users to see a "Public Image" field on a Node.
2. The Action: You grant the "View media" (or "View published content") permission to Anonymous users.
3. The Consequence: Because the user can now "View" the entity, Drupal automatically grants them download access to ALL files attached to that entity—including your "Private Document" field.

To solve this, we must intervene at the file download level (the Controller) to override this global permission logic.

Technical Obstacles: Why Standard Solutions Failed

We encountered five distinct technical blockers that dictated this specific architecture.

Part 1: Routing & Security Blockers (Why we use Base64)

1. The "s3://" Protocol Gap Browsers do not understand s3://folder/doc.pdf. Normally, S3FS translates this to a direct URL. However, because our bucket is Private, direct links return 403 Forbidden. We must proxy the traffic.

2. The Failure of Query Parameters We cannot use a simple proxy URL like /s3-proxy?file=secure/doc.pdf because:

• Security False-Positives: URLs containing file paths in query strings trigger Local File Inclusion (LFI) alerts in Web Application Firewalls (WAF), blocking legitimate users.
• Caching Issues: CDNs often strip query parameters, breaking file delivery.

3. The Routing "Slash" Problem We cannot use a clean path like /s3-proxy/secure/docs/file.pdf because Drupal's router interprets the slashes as separate arguments. Using Regex to fix this re-introduces URL encoding bugs and security filter issues.

The Solution: By encoding the path into a Base64 string, we present Drupal with a single, safe alphanumeric string (e.g., Zm9sZGVyL2ZpbGUucGRm) that bypasses all WAF filters and routing issues.

Part 2: Module & Caching Blockers (Why we use Custom Folders & Controller)

4. The S3FS Reserved Keyword Crash If you name your S3 folders public/ or private/, the S3FS module throws a CrossSchemeAccessException. It mistakenly thinks you are trying to spoof the native file system.

• The Fix: We use synonyms: open/ for public files and file-private/ for private files.

5. The CNAME / Presigned URL Problem Standard S3 "Presigned URLs" contain a unique signature for every user session. This breaks CDN caching because the URL is never the same twice. Our Proxy solution creates stable URLs that CDNs can cache permanently.

The Architecture: "The Base64 Proxy"

We will use the native s3:// scheme to keep this site's files isolated.

1. Storage (s3://): All files live in the private S3 bucket.
2. Routing (The Rewrite): We use a hook to rewrite all s3:// URLs to point to our custom route: /s3-proxy/{base64_key}.
3. Access Control: The Controller decodes the key and checks the folder name:
   • /open/ Folder: Explicitly ALLOW access (and set Public Cache headers).
   • /file-private/ Folder: Explicitly DENY access unless logged in (and set Private Cache headers).

Step 1: The Configuration

First, configure S3fs to use the native scheme.

S3fs Settings (/admin/config/media/s3fs):

• Use S3 for public:// files: Uncheck (Keep local).
• Use S3 for private:// files: Uncheck (Keep local).
• S3 Stream Wrapper: Enable (Activates s3://).
• AWS Bucket Settings: "Block All Public Access" remains ON.

Field & CKEditor Settings: For every file field or CKEditor Image Upload setting:

• Upload Destination: Set to S3 File System.
• File Directory: MUST start with open (e.g., open/images) or file-private (e.g., file-private/docs). Do NOT use public or private.

Step 2: The Routing

We need a route that accepts the Base64 string. Create [CUSTOM_MODULE].routing.yml:

[CUSTOM_MODULE].s3_proxy:
  path: '/s3-proxy/{key}'
  defaults:
    _controller: '\Drupal\[CUSTOM_MODULE]\Controller\S3ProxyController::download'
    _title: 'File Download'
  requirements:
    # Allow everyone to reach the controller; logic inside handles the denial.
    _access: 'TRUE'
    # Allow the key to contain any characters.
    key: .+
  options:
    # Disable internal page cache for this route to ensure permissions run
    no_cache: 'TRUE'

Step 3: The Controller (The Brain)

This controller handles decoding, permission checking (file-private vs open), streaming, and auto-generating missing thumbnails.

Features:

• Locking: Prevents race conditions during thumbnail generation.
• Caching: Sets Cache-Control: public for open/ files (enabling CDN) and private for file-private/ files.

Create src/Controller/S3ProxyController.php:

<?php

namespace Drupal\odb_core\Controller;

use Aws\S3\Exception\S3Exception;
use Drupal\Core\Controller\ControllerBase;
use Drupal\Core\DependencyInjection\ContainerInjectionInterface;
use Drupal\Core\Entity\EntityTypeManagerInterface;
use Drupal\Core\Lock\LockBackendInterface;
use Drupal\Core\Logger\LoggerChannelFactoryInterface;
use Drupal\Core\Session\AccountProxyInterface;
use Drupal\s3fs\S3fsService;
use Symfony\Component\DependencyInjection\ContainerInterface;
use Symfony\Component\HttpFoundation\RequestStack;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\HttpFoundation\StreamedResponse;
use Symfony\Component\HttpKernel\Exception\AccessDeniedHttpException;
use Symfony\Component\HttpKernel\Exception\NotFoundHttpException;

class S3ProxyController extends ControllerBase implements ContainerInjectionInterface {

  protected $loggerFactory;
  protected $requestStack;
  protected $currentUser;
  protected $s3fs;
  protected $entityTypeManager;
  protected $lock;

  public function __construct(
    LoggerChannelFactoryInterface $logger_factory,
    RequestStack $request_stack,
    AccountProxyInterface $current_user,
    S3fsService $s3fs,
    EntityTypeManagerInterface $entity_type_manager,
    LockBackendInterface $lock
  ) {
    $this->loggerFactory = $logger_factory;
    $this->requestStack = $request_stack;
    $this->currentUser = $current_user;
    $this->s3fs = $s3fs;
    $this->entityTypeManager = $entity_type_manager;
    $this->lock = $lock;
  }

  public static function create(ContainerInterface $container) {
    return new static(
      $container->get('logger.factory'),
      $container->get('request_stack'),
      $container->get('current_user'),
      $container->get('s3fs'),
      $container->get('entity_type.manager'),
      $container->get('lock')
    );
  }

  public function download($key) {
    // 1. Decode Key
    $decoded_key = base64_decode($key);

    // 2. Hybrid Access Control
    // Logic: If path contains "file-private/", user must be logged in.
    $is_secure_file = (strpos($decoded_key, 'file-private/') === 0) || (strpos($decoded_key, '/file-private/') !== FALSE);

    if ($is_secure_file && $this->currentUser->isAnonymous()) {
      throw new AccessDeniedHttpException('Secure file access denied.');
    }

    // 3. Get S3 Config
    $s3fs_config = $this->config('s3fs.settings')->get();
    $bucket = $s3fs_config['bucket'] ?? NULL;

    if (empty($bucket)) {
      throw new NotFoundHttpException();
    }

    // 4. Init Client
    try {
      $s3_client = $this->s3fs->getAmazonS3Client($s3fs_config);
    }
    catch (\Exception $e) {
      $this->loggerFactory->get('odb_core')->error('S3 Client Error: @m', ['@m' => $e->getMessage()]);
      throw new NotFoundHttpException();
    }

    // 5. Verify Object & Handle Missing Thumbnails
    if (!$s3_client->doesObjectExist($bucket, $decoded_key)) {
      // CHECK: Is this a request for a missing Image Style (thumbnail)?
      if (strpos($decoded_key, 'styles/') === 0) {
        if (!$this->generateImageStyle($decoded_key, $s3_client, $bucket)) {
           return new Response('Error: Image style generation failed.', 404);
        }
        // If success, we proceed to stream the newly created file below!
      } else {
        return new Response('Error: File not found.', 404);
      }
    }

    // 6. Stream Response
    $response = new StreamedResponse(function () use ($s3_client, $bucket, $decoded_key) {
      try {
        $result = $s3_client->getObject(['Bucket' => $bucket, 'Key' => $decoded_key]);
        if ($result['Body']) echo $result['Body'];
      }
      catch (S3Exception $e) {}
    });

    // 7. Headers & Cache Control
    try {
      $head = $s3_client->headObject(['Bucket' => $bucket, 'Key' => $decoded_key]);
      $response->headers->set('Content-Type', $head['ContentType']);
      $response->headers->set('Content-Length', $head['ContentLength']);
      $response->headers->set('Content-Disposition', 'inline; filename="' . basename($decoded_key) . '"');

      if ($is_secure_file) {
        // Secure Files: Browser only, short life. Prevent CDN caching.
        $response->headers->set('Cache-Control', 'private, max-age=3600');
      }
      else {
        // Open Files: Cache in CDN/Cloudflare.
        // 'public' directive enables shared caching.
        $response->headers->set('Cache-Control', 'public, max-age=604800, s-maxage=2592000');
      }
    }
    catch (\Exception $e) {}

    return $response;
  }

  private function generateImageStyle($key, $s3_client, $bucket) {
    $parts = explode('/', $key);
    // Structure: styles/{style}/s3/{path}
    if (count($parts) < 4 || $parts[0] !== 'styles' || $parts[2] !== 's3') return FALSE;

    $style_name = $parts[1];
    $source_path = implode('/', array_slice($parts, 3)); 
    $source_uri = 's3://' . $source_path;

    // Handle .webp extension mismatch
    $real_source_uri = $source_uri;
    if (str_ends_with($source_uri, '.webp')) {
       $without_webp = substr($source_uri, 0, -5);
       if (pathinfo($without_webp, PATHINFO_EXTENSION)) {
         $real_source_uri = $without_webp;
       }
    }

    // Lock to prevent race conditions
    $lock_name = 's3_proxy_generate:' . $key;
    if (!$this->lock->acquire($lock_name)) {
      $this->lock->wait($lock_name);
      if ($s3_client->doesObjectExist($bucket, $key)) return TRUE;
      if (!$this->lock->acquire($lock_name)) return FALSE;
    }
    
    try {
      $style = $this->entityTypeManager->getStorage('image_style')->load($style_name);
      $result = $style ? $style->createDerivative($real_source_uri, 's3://' . $key) : FALSE;
    }
    catch (\Exception $e) {
      $result = FALSE;
    }

    $this->lock->release($lock_name);
    return $result;
  }
}

Step 4: The Global Rewriter (The Automator)

We use hook_file_url_alter to intercept every s3:// URL generated by Drupal and rewrite it to point to our Proxy Route with the Base64 key.

Add this to [CUSTOM_MODULE].module:

use Drupal\Core\Url;

/**
 * Implements hook_file_url_alter().
 */
function [CUSTOM_MODULE]_file_url_alter(&$uri) {
  // Check if the URI scheme is 's3'
  $scheme = \Drupal::service('stream_wrapper_manager')->getScheme($uri);

  if ($scheme === 's3') {
    // Extract the relative path (the "Key")
    $target = \Drupal::service('stream_wrapper_manager')->getTarget($uri);
    
    // 1. Base64 Encode the path
    $encoded_key = base64_encode($target);

    // 2. Generate the Proxy URL
    $url = Url::fromRoute('[CUSTOM_MODULE].s3_proxy', ['key' => $encoded_key]);
    
    // 3. Replace the original URI
    $uri = $url->toString();
  }
}

Step 5: Enforcing Folder Structure

Update the validator to ensure no one accidentally uses "public" or "private".

use Drupal\Core\Form\FormStateInterface;
use Drupal\Core\Entity\EntityFormInterface;

/**
 * Implements hook_form_FORM_ID_alter() for 'field_config_edit_form'.
 */
function [CUSTOM_MODULE]_form_field_config_edit_form_alter(array &$form, FormStateInterface $form_state) {
  $form_object = $form_state->getFormObject();
  if (!$form_object instanceof EntityFormInterface) return;

  $field_config = $form_object->getEntity();
  if (!in_array($field_config->getType(), ['file', 'image'])) return;

  // Force S3 scheme
  if (isset($form['field_storage']['subform']['settings']['uri_scheme'])) {
    $form['field_storage']['subform']['settings']['uri_scheme']['#default_value'] = 's3';
    $form['field_storage']['subform']['settings']['uri_scheme']['#disabled'] = TRUE;
  }

  // Validate directory prefix
  $form['#validate'][] = '_[CUSTOM_MODULE]_validate_file_directory';
}

function _[CUSTOM_MODULE]_validate_file_directory(array &$form, FormStateInterface $form_state) {
  $directory = $form_state->getValue(['settings', 'file_directory']);
  
  // [UPDATED] Enforce 'open' or 'file-private' to prevent S3FS crashes
  if (empty($directory) || !preg_match('/^(open|file-private)(\/|$)/', $directory)) {
    $form_state->setErrorByName('settings][file_directory', t('Directory must start with "open" or "file-private". Do NOT use "public" or "private".'));
  }
}

Troubleshooting Checklist

1. Clear Cache: Run drush cr immediately after deploying.
2. S3FS Crash: If you see CrossSchemeAccessException, verify you changed your folder names to open/ (optional) and file-private/ (must-have, to prevent clashes) in every field and CKEditor profile.
3. Redirect Loops: If image styles loop indefinitely, check the Drupal log (/admin/reports/dblog). It usually means the Controller failed to find the source image to generate the thumbnail.