How to Write AWS Glue File Output to a Specific Name: A Complete Guide (2026)

In AWS Glue, when you run a job that processes data and writes the output to an S3 bucket, the files are often named with a generic pattern like run-123456789-part-r-00000. While this default naming convention is functional, it might not suit all business needs. This tutorial will guide you through the steps to customize your output file names, such as renaming them to a more meaningful format like Customer_Transaction.json.

Key Takeaways

• Understand how AWS Glue jobs name output files by default.
• Learn to customize output filenames using AWS Glue and PySpark.
• Implement the solution in a practical AWS Glue job scenario.
• Learn common pitfalls and troubleshooting tips.

When working with AWS Glue, a serverless ETL service, you have the power of Apache Spark to transform and move data efficiently. However, customizing the output filenames can be a bit tricky due to the distributed nature of Spark operations. This guide will help you understand the intricacies of handling file outputs in AWS Glue and provide a step-by-step method to achieve your desired output naming convention. This is particularly useful for scenarios where you need to manage data pipelines with specific file naming requirements for downstream applications.

Prerequisites

• Basic understanding of AWS Glue and its components.
• Familiarity with PySpark and Python programming.
• Access to an AWS account with permissions to create and run Glue jobs.
• An existing AWS Glue job that reads from a source (e.g., Aurora tables) and writes to an S3 bucket.

Step 1: Understand Default File Naming in AWS Glue

AWS Glue uses Apache Spark, which by default, writes output files in a distributed manner. This often results in filenames such as run-123456789-part-r-00000. These names are generated based on the Spark job's execution context, which splits data across multiple partitions and writes them in parallel. To customize these filenames, we need to intercept the data before it is written to S3 and manage the writing process manually.

Step 2: Customize the Output Filename Using PySpark

The key to customizing output filenames in AWS Glue is to collect the data into a single partition and then write it with your desired filename. This can be achieved using PySpark's DataFrame operations.

from pyspark.sql import SparkSession

# Initialize Spark session
def get_spark_session():
    return SparkSession.builder.appName("AWS Glue Custom Output Filename")\
        .config("spark.sql.warehouse.dir", "./")\
        .enableHiveSupport()\
        .getOrCreate()

spark = get_spark_session()

# Example DataFrame
customers_df = spark.read.json("s3://your-bucket/input/customers.json")
transactions_df = spark.read.json("s3://your-bucket/input/transactions.json")

# Perform the join operation
joined_df = customers_df.join(transactions_df, customers_df.customer_id == transactions_df.customer_id)

# Coalesce the DataFrame to a single partition
single_partition_df = joined_df.coalesce(1)

# Write to S3 with a specific filename
single_partition_df.write.mode("overwrite").json("s3://your-bucket/output/temp/")

By coalescing the DataFrame down to a single partition, you ensure that all data is written to a single file, which you can then rename in the subsequent step.

Step 3: Rename the Output File in S3

After writing the file to a temporary location in your S3 bucket, you need to rename it to your desired filename using the AWS SDK for Python (Boto3).

import boto3

# Boto3 S3 client
s3_client = boto3.client('s3')

# Define the bucket and temporary output path
bucket_name = 'your-bucket'
temp_output_path = 'output/temp/'
final_output_filename = 'output/Customer_Transaction.json'

# List objects in the temporary output directory
objects = s3_client.list_objects_v2(Bucket=bucket_name, Prefix=temp_output_path)

# Identify the newly written file
for obj in objects.get('Contents', []):
    if obj['Key'].endswith('.json'):
        # Copy the object to the final location with the desired filename
        copy_source = {'Bucket': bucket_name, 'Key': obj['Key']}
        s3_client.copy_object(CopySource=copy_source, Bucket=bucket_name, Key=final_output_filename)

        # Optionally delete the temporary files
        s3_client.delete_object(Bucket=bucket_name, Key=obj['Key'])

This script will copy the file from the temporary location to your desired path with the specified filename, thereby achieving the renaming.

Common Errors/Troubleshooting

• FileNotFoundError: Ensure the temporary output path is correct and objects are present.
• Insufficient Permissions: Verify that your AWS IAM role has the necessary permissions for S3 read/write operations.
• Performance Issues: Coalescing can be resource-intensive; ensure your Glue job is adequately provisioned.

Frequently Asked Questions

Why do I need to coalesce the DataFrame?

Coalescing the DataFrame reduces it to a single partition, allowing the data to be written as one file, which you can then rename.

Can I use other file formats besides JSON?

Yes, you can use formats like CSV or Parquet by changing the write method accordingly.

Is this method suitable for large datasets?

For very large datasets, consider the performance implications of coalescing and ensure your Glue job has sufficient resources.

Frequently Asked Questions

Why do I need to coalesce the DataFrame?

Coalescing the DataFrame reduces it to a single partition, allowing the data to be written as one file, which you can then rename.

Can I use other file formats besides JSON?

Yes, you can use formats like CSV or Parquet by changing the write method accordingly.

Is this method suitable for large datasets?

For very large datasets, consider the performance implications of coalescing and ensure your Glue job has sufficient resources.