I am currently working on a Jenkins declarative pipeline to connect the Jenkins builds with Kubernetes, Helm and Netflix Spinnaker. One of TODOs has been to deploy different artifacts (e.g. a helm chart my-chart-0.0.1.tar.gz) to an AWS S3-compatible bucket inside a Minio installation with help of pipeline-aws-plugin.
When running
withAWS(endpointUrl: 'https://minio.domain.tld', credentials: config.credentialsId) { s3Upload(file: "my-file.txt", bucket: "my-bucket") }
my pipeline always threw an exception with
com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: null; S3 Extended Request ID: null), S3 Extended Request ID: null at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1695) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1350) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1101) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:758) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:732) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:714) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:674) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:656) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:520) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4705) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4652)
Trying other clients with Minio
At first I suspected some misconfiguration of my Minio installation. I checked the S3 upload with mc and AWS’ own CLI. Both worked flawlessly so it had to be something else.
Enable logging
To get some more debugging output, I configured Jenkins’ to log events for com.amazonaws and org.apache.http.wire. The debugging output does not show up inside the build job’s console output but under the configured logger.
Host-style access to S3 buckets
After scanning the debug output, I noticed the following:
http-outgoing-11 >> "PUT /my-file.txt HTTP/1.1[\r][\n]" Jan 21, 2019 9:36:15 PM FINE org.apache.http.impl.conn.Wire wire http-outgoing-11 >> "Host: my-bucket.minio.domain.tld[\r][\n]" Jan 21, 2019 9:36:15 PM FINE org.apache.http.impl.conn.Wire wire http-outgoing-11 >> "x-amz-content-sha256: UNSIGNED-PAYLOAD[\r][\n]" ... http-outgoing-11 << "[\r][\n]" Jan 21, 2019 9:36:15 PM FINE org.apache.http.impl.conn.Wire wire http-outgoing-11 << "default backend - 404" Jan 21, 2019 9:36:15 PM FINE com.amazonaws.services.s3.internal.S3ErrorResponseHandler createException Failed in parsing the error response : default backend - 404
When pipeline-aws-plugin initiates a request to my bucket, it does not https://minio.domain.tld/my-bucket but https://my-bucket.minio.domain.tld. This is totally fine for AWS S3 buckets. But with the Minio deployment in our Kubernetes cluster, this does not work out of the box:
- By default, our Minio deployment does not use the –address parameter described in https://github.com/minio/minio/issues/4681
- Our Minio ingress does also not listen to 4th level domains like my-bucket.minio.domain.tld so that the nginx proxy returns the “default backend – 404” string seen in the log output above.
Solving the issue
Instead of configuring host-style access I fixed it by simply using the pathStyleAccessEnabled: true in my s3Upload step. When enabled, pipeline-aws-plugin does not use the bucket name as 4th level subdomain but appends the bucket name to the host name:
withAWS(endpointUrl: 'https://minio.domain.tld', credentials: config.credentialsId) { s3Upload(pathStyleAccessEnabled: true, file: "my-file.txt", bucket: "my-bucket") }