feat: V2 microservices stack — backend services, gateway, JWT auth
Add full V2 architecture: identity, content, studio (.NET 10) and file, render, notification, gateway (Go) services with vendored deps, plus DB migrations, event/API contracts, and an init-db script. Wire the Next.js frontend to the gateway: server-side JWT auth routes (login/register/refresh/logout/me), gateway fetch helper, and session/ cookie/jwt helpers under src/lib. Containerize the stack via docker-compose.v2.yml and per-service Dockerfiles. Base images resolve through a Nexus mirror (Docker Hub) and MCR directly; npm/NuGet pull from Nexus groups. Self-host fonts via next/font/local to avoid Google Fonts (geo-blocked). Add CI workflow and ignore .env.v2, *.stackdump, and .NET bin/obj. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
soroush.asadi
+202
@@ -0,0 +1,202 @@
|
||||
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
+27
@@ -0,0 +1,27 @@
|
||||
Copyright (c) 2009 The Go Authors. All rights reserved.
|
||||
|
||||
Redistribution and use in source and binary forms, with or without
|
||||
modification, are permitted provided that the following conditions are
|
||||
met:
|
||||
|
||||
* Redistributions of source code must retain the above copyright
|
||||
notice, this list of conditions and the following disclaimer.
|
||||
* Redistributions in binary form must reproduce the above
|
||||
copyright notice, this list of conditions and the following disclaimer
|
||||
in the documentation and/or other materials provided with the
|
||||
distribution.
|
||||
* Neither the name of Google Inc. nor the names of its
|
||||
contributors may be used to endorse or promote products derived from
|
||||
this software without specific prior written permission.
|
||||
|
||||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
|
||||
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
|
||||
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
|
||||
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
|
||||
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
|
||||
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
|
||||
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
|
||||
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
|
||||
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
||||
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
||||
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
||||
+198
@@ -0,0 +1,198 @@
|
||||
|
||||
# md5-simd
|
||||
|
||||
This is a SIMD accelerated MD5 package, allowing up to either 8 (AVX2) or 16 (AVX512) independent MD5 sums to be calculated on a single CPU core.
|
||||
|
||||
It was originally based on the [md5vec](https://github.com/igneous-systems/md5vec) repository by Igneous Systems, but has been made more flexible by amongst others supporting different message sizes per lane and adding AVX512.
|
||||
|
||||
`md5-simd` integrates a similar mechanism as described in [minio/sha256-simd](https://github.com/minio/sha256-simd#support-for-avx512) for making it easy for clients to take advantages of the parallel nature of the MD5 calculation. This will result in reduced overall CPU load.
|
||||
|
||||
It is important to understand that `md5-simd` **does not speed up** a single threaded MD5 hash sum.
|
||||
Rather it allows multiple __independent__ MD5 sums to be computed in parallel on the same CPU core,
|
||||
thereby making more efficient usage of the computing resources.
|
||||
|
||||
## Usage
|
||||
|
||||
[](https://pkg.go.dev/github.com/minio/md5-simd?tab=doc)
|
||||
|
||||
|
||||
In order to use `md5-simd`, you must first create an `Server` which can be
|
||||
used to instantiate one or more objects for MD5 hashing.
|
||||
|
||||
These objects conform to the regular [`hash.Hash`](https://pkg.go.dev/hash?tab=doc#Hash) interface
|
||||
and as such the normal Write/Reset/Sum functionality works as expected.
|
||||
|
||||
As an example:
|
||||
```
|
||||
// Create server
|
||||
server := md5simd.NewServer()
|
||||
defer server.Close()
|
||||
|
||||
// Create hashing object (conforming to hash.Hash)
|
||||
md5Hash := server.NewHash()
|
||||
defer md5Hash.Close()
|
||||
|
||||
// Write one (or more) blocks
|
||||
md5Hash.Write(block)
|
||||
|
||||
// Return digest
|
||||
digest := md5Hash.Sum([]byte{})
|
||||
```
|
||||
|
||||
To keep performance both a [Server](https://pkg.go.dev/github.com/minio/md5-simd?tab=doc#Server)
|
||||
and individual [Hasher](https://pkg.go.dev/github.com/minio/md5-simd?tab=doc#Hasher) should
|
||||
be closed using the `Close()` function when no longer needed.
|
||||
|
||||
A Hasher can efficiently be re-used by using [`Reset()`](https://pkg.go.dev/hash?tab=doc#Hash) functionality.
|
||||
|
||||
In case your system does not support the instructions required it will fall back to using `crypto/md5` for hashing.
|
||||
|
||||
## Limitations
|
||||
|
||||
As explained above `md5-simd` does not speed up an individual MD5 hash sum computation,
|
||||
unless some hierarchical tree construct is used but this will result in different outcomes.
|
||||
Running a single hash on a server results in approximately half the throughput.
|
||||
|
||||
Instead, it allows running multiple MD5 calculations in parallel on a single CPU core.
|
||||
This can be beneficial in e.g. multi-threaded server applications where many go-routines
|
||||
are dealing with many requests and multiple MD5 calculations can be packed/scheduled for parallel execution on a single core.
|
||||
|
||||
This will result in a lower overall CPU usage as compared to using the standard `crypto/md5`
|
||||
functionality where each MD5 hash computation will consume a single thread (core).
|
||||
|
||||
It is best to test and measure the overall CPU usage in a representative usage scenario in your application
|
||||
to get an overall understanding of the benefits of `md5-simd` as compared to `crypto/md5`, ideally under heavy CPU load.
|
||||
|
||||
Also note that `md5-simd` is best meant to work with large objects,
|
||||
so if your application only hashes small objects of a few kilobytes
|
||||
you may be better of by using `crypto/md5`.
|
||||
|
||||
## Performance
|
||||
|
||||
For the best performance writes should be a multiple of 64 bytes, ideally a multiple of 32KB.
|
||||
To help with that a [`buffered := bufio.NewWriterSize(hasher, 32<<10)`](https://golang.org/pkg/bufio/#NewWriterSize)
|
||||
can be inserted if you are unsure of the sizes of the writes.
|
||||
Remember to [flush](https://golang.org/pkg/bufio/#Writer.Flush) `buffered` before reading the hash.
|
||||
|
||||
A single 'server' can process 16 streams concurrently with 1 core (AVX-512) or 2 cores (AVX2).
|
||||
In situations where it is likely that more than 16 streams are fully loaded it may be beneficial
|
||||
to use multiple servers.
|
||||
|
||||
The following chart compares the multi-core performance between `crypto/md5` vs the AVX2 vs the AVX512 code:
|
||||
|
||||
|
||||
|
||||
Compared to `crypto/md5`, the AVX2 version is up to 4x faster:
|
||||
|
||||
```
|
||||
$ benchcmp crypto-md5.txt avx2.txt
|
||||
benchmark old MB/s new MB/s speedup
|
||||
BenchmarkParallel/32KB-4 2229.22 7370.50 3.31x
|
||||
BenchmarkParallel/64KB-4 2233.61 8248.46 3.69x
|
||||
BenchmarkParallel/128KB-4 2235.43 8660.74 3.87x
|
||||
BenchmarkParallel/256KB-4 2236.39 8863.87 3.96x
|
||||
BenchmarkParallel/512KB-4 2238.05 8985.39 4.01x
|
||||
BenchmarkParallel/1MB-4 2233.56 9042.62 4.05x
|
||||
BenchmarkParallel/2MB-4 2224.11 9014.46 4.05x
|
||||
BenchmarkParallel/4MB-4 2199.78 8993.61 4.09x
|
||||
BenchmarkParallel/8MB-4 2182.48 8748.22 4.01x
|
||||
```
|
||||
|
||||
Compared to `crypto/md5`, the AVX512 is up to 8x faster (for larger block sizes):
|
||||
|
||||
```
|
||||
$ benchcmp crypto-md5.txt avx512.txt
|
||||
benchmark old MB/s new MB/s speedup
|
||||
BenchmarkParallel/32KB-4 2229.22 11605.78 5.21x
|
||||
BenchmarkParallel/64KB-4 2233.61 14329.65 6.42x
|
||||
BenchmarkParallel/128KB-4 2235.43 16166.39 7.23x
|
||||
BenchmarkParallel/256KB-4 2236.39 15570.09 6.96x
|
||||
BenchmarkParallel/512KB-4 2238.05 16705.83 7.46x
|
||||
BenchmarkParallel/1MB-4 2233.56 16941.95 7.59x
|
||||
BenchmarkParallel/2MB-4 2224.11 17136.01 7.70x
|
||||
BenchmarkParallel/4MB-4 2199.78 17218.61 7.83x
|
||||
BenchmarkParallel/8MB-4 2182.48 17252.88 7.91x
|
||||
```
|
||||
|
||||
These measurements were performed on AWS EC2 instance of type `c5.xlarge` equipped with a Xeon Platinum 8124M CPU at 3.0 GHz.
|
||||
|
||||
If only one or two inputs are available the scalar calculation method will be used for the
|
||||
optimal speed in these cases.
|
||||
|
||||
## Operation
|
||||
|
||||
To make operation as easy as possible there is a “Server” coordinating everything. The server keeps track of individual hash states and updates them as new data comes in. This can be visualized as follows:
|
||||
|
||||
|
||||
|
||||
The data is sent to the server from each hash input in blocks of up to 32KB per round. In our testing we found this to be the block size that yielded the best results.
|
||||
|
||||
Whenever there is data available the server will collect data for up to 16 hashes and process all 16 lanes in parallel. This means that if 16 hashes have data available all the lanes will be filled. However since that may not be the case, the server will fill less lanes and do a round anyway. Lanes can also be partially filled if less than 32KB of data is written.
|
||||
|
||||
|
||||
|
||||
In this example 4 lanes are fully filled and 2 lanes are partially filled. In this case the black areas will simply be masked out from the results and ignored. This is also why calculating a single hash on a server will not result in any speedup and hash writes should be a multiple of 32KB for the best performance.
|
||||
|
||||
For AVX512 all 16 calculations will be done on a single core, on AVX2 on 2 cores if there is data for more than 8 lanes.
|
||||
So for optimal usage there should be data available for all 16 hashes. It may be perfectly reasonable to use more than 16 concurrent hashes.
|
||||
|
||||
|
||||
## Design & Tech
|
||||
|
||||
md5-simd has both an AVX2 (8-lane parallel), and an AVX512 (16-lane parallel version) algorithm to accelerate the computation with the following function definitions:
|
||||
```
|
||||
//go:noescape
|
||||
func block8(state *uint32, base uintptr, bufs *int32, cache *byte, n int)
|
||||
|
||||
//go:noescape
|
||||
func block16(state *uint32, ptrs *int64, mask uint64, n int)
|
||||
```
|
||||
|
||||
The AVX2 version is based on the [md5vec](https://github.com/igneous-systems/md5vec) repository and is essentially unchanged except for minor (cosmetic) changes.
|
||||
|
||||
The AVX512 version is derived from the AVX2 version but adds some further optimizations and simplifications.
|
||||
|
||||
### Caching in upper ZMM registers
|
||||
|
||||
The AVX2 version passes in a `cache8` block of memory (about 0.5 KB) for temporary storage of intermediate results during `ROUND1` which are subsequently used during `ROUND2` through to `ROUND4`.
|
||||
|
||||
Since AVX512 has double the amount of registers (32 ZMM registers as compared to 16 YMM registers), it is possible to use the upper 16 ZMM registers for keeping the intermediate states on the CPU. As such, there is no need to pass in a corresponding `cache16` into the AVX512 block function.
|
||||
|
||||
### Direct loading using 64-bit pointers
|
||||
|
||||
The AVX2 uses the `VPGATHERDD` instruction (for YMM) to do a parallel load of 8 lanes using (8 independent) 32-bit offets. Since there is no control over how the 8 slices that are passed into the (Golang) `blockMd5` function are laid out into memory, it is not possible to derive a "base" address and corresponding offsets (all within 32-bits) for all 8 slices.
|
||||
|
||||
As such the AVX2 version uses an interim buffer to collect the byte slices to be hashed from all 8 inut slices and passed this buffer along with (fixed) 32-bit offsets into the assembly code.
|
||||
|
||||
For the AVX512 version this interim buffer is not needed since the AVX512 code uses a pair of `VPGATHERQD` instructions to directly dereference 64-bit pointers (from a base register address that is initialized to zero).
|
||||
|
||||
Note that two load (gather) instructions are needed because the AVX512 version processes 16-lanes in parallel, requiring 16 times 64-bit = 1024 bits in total to be loaded. A simple `VALIGND` and `VPORD` are subsequently used to merge the lower and upper halves together into a single ZMM register (that contains 16 lanes of 32-bit DWORDS).
|
||||
|
||||
### Masking support
|
||||
|
||||
Due to the fact that pointers are passed directly from the Golang slices, we need to protect against NULL pointers.
|
||||
For this a 16-bit mask is passed in the AVX512 assembly code which is used during the `VPGATHERQD` instructions to mask out lanes that could otherwise result in segment violations.
|
||||
|
||||
### Minor optimizations
|
||||
|
||||
The `roll` macro (three instructions on AVX2) is no longer needed for AVX512 and is replaced by a single `VPROLD` instruction.
|
||||
|
||||
Also several logical operations from the various ROUNDS of the AVX2 version could be combined into a single instruction using ternary logic (with the `VPTERMLOGD` instruction), resulting in a further simplification and speed-up.
|
||||
|
||||
## Low level block function performance
|
||||
|
||||
The benchmark below shows the (single thread) maximum performance of the `block()` function for AVX2 (having 8 lanes) and AVX512 (having 16 lanes). Also the baseline single-core performance from the standard `crypto/md5` package is shown for comparison.
|
||||
|
||||
```
|
||||
BenchmarkCryptoMd5-4 687.66 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkBlock8-4 4144.80 MB/s 0 B/op 0 allocs/op
|
||||
BenchmarkBlock16-4 8228.88 MB/s 0 B/op 0 allocs/op
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
`md5-simd` is released under the Apache License v2.0. You can find the complete text in the file LICENSE.
|
||||
|
||||
## Contributing
|
||||
|
||||
Contributions are welcome, please send PRs for any enhancements.
|
||||
+228
@@ -0,0 +1,228 @@
|
||||
// Copyright (c) 2020 MinIO Inc. All rights reserved.
|
||||
// Use of this source code is governed by a license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
//+build !noasm,!appengine,gc
|
||||
|
||||
// This is the AVX512 implementation of the MD5 block function (16-way parallel)
|
||||
|
||||
#define prep(index) \
|
||||
KMOVQ kmask, ktmp \
|
||||
VPGATHERDD index*4(base)(ptrs*1), ktmp, mem
|
||||
|
||||
#define ROUND1(a, b, c, d, index, const, shift) \
|
||||
VPXORQ c, tmp, tmp \
|
||||
VPADDD 64*const(consts), a, a \
|
||||
VPADDD mem, a, a \
|
||||
VPTERNLOGD $0x6C, b, d, tmp \
|
||||
prep(index) \
|
||||
VPADDD tmp, a, a \
|
||||
VPROLD $shift, a, a \
|
||||
VMOVAPD c, tmp \
|
||||
VPADDD b, a, a
|
||||
|
||||
#define ROUND1noload(a, b, c, d, const, shift) \
|
||||
VPXORQ c, tmp, tmp \
|
||||
VPADDD 64*const(consts), a, a \
|
||||
VPADDD mem, a, a \
|
||||
VPTERNLOGD $0x6C, b, d, tmp \
|
||||
VPADDD tmp, a, a \
|
||||
VPROLD $shift, a, a \
|
||||
VMOVAPD c, tmp \
|
||||
VPADDD b, a, a
|
||||
|
||||
#define ROUND2(a, b, c, d, zreg, const, shift) \
|
||||
VPADDD 64*const(consts), a, a \
|
||||
VPADDD zreg, a, a \
|
||||
VANDNPD c, tmp, tmp \
|
||||
VPTERNLOGD $0xEC, b, tmp, tmp2 \
|
||||
VMOVAPD c, tmp \
|
||||
VPADDD tmp2, a, a \
|
||||
VMOVAPD c, tmp2 \
|
||||
VPROLD $shift, a, a \
|
||||
VPADDD b, a, a
|
||||
|
||||
#define ROUND3(a, b, c, d, zreg, const, shift) \
|
||||
VPADDD 64*const(consts), a, a \
|
||||
VPADDD zreg, a, a \
|
||||
VPTERNLOGD $0x96, b, d, tmp \
|
||||
VPADDD tmp, a, a \
|
||||
VPROLD $shift, a, a \
|
||||
VMOVAPD b, tmp \
|
||||
VPADDD b, a, a
|
||||
|
||||
#define ROUND4(a, b, c, d, zreg, const, shift) \
|
||||
VPADDD 64*const(consts), a, a \
|
||||
VPADDD zreg, a, a \
|
||||
VPTERNLOGD $0x36, b, c, tmp \
|
||||
VPADDD tmp, a, a \
|
||||
VPROLD $shift, a, a \
|
||||
VPXORQ c, ones, tmp \
|
||||
VPADDD b, a, a
|
||||
|
||||
TEXT ·block16(SB), 4, $0-40
|
||||
|
||||
MOVQ state+0(FP), BX
|
||||
MOVQ base+8(FP), SI
|
||||
MOVQ ptrs+16(FP), AX
|
||||
KMOVQ mask+24(FP), K1
|
||||
MOVQ n+32(FP), DX
|
||||
MOVQ ·avx512md5consts+0(SB), DI
|
||||
|
||||
#define a Z0
|
||||
#define b Z1
|
||||
#define c Z2
|
||||
#define d Z3
|
||||
|
||||
#define sa Z4
|
||||
#define sb Z5
|
||||
#define sc Z6
|
||||
#define sd Z7
|
||||
|
||||
#define tmp Z8
|
||||
#define tmp2 Z9
|
||||
#define ptrs Z10
|
||||
#define ones Z12
|
||||
#define mem Z15
|
||||
|
||||
#define kmask K1
|
||||
#define ktmp K3
|
||||
|
||||
// ----------------------------------------------------------
|
||||
// Registers Z16 through to Z31 are used for caching purposes
|
||||
// ----------------------------------------------------------
|
||||
|
||||
#define dig BX
|
||||
#define count DX
|
||||
#define base SI
|
||||
#define consts DI
|
||||
|
||||
// load digest into state registers
|
||||
VMOVUPD (dig), a
|
||||
VMOVUPD 0x40(dig), b
|
||||
VMOVUPD 0x80(dig), c
|
||||
VMOVUPD 0xc0(dig), d
|
||||
|
||||
// load source pointers
|
||||
VMOVUPD 0x00(AX), ptrs
|
||||
|
||||
MOVQ $-1, AX
|
||||
VPBROADCASTQ AX, ones
|
||||
|
||||
loop:
|
||||
VMOVAPD a, sa
|
||||
VMOVAPD b, sb
|
||||
VMOVAPD c, sc
|
||||
VMOVAPD d, sd
|
||||
|
||||
prep(0)
|
||||
VMOVAPD d, tmp
|
||||
VMOVAPD mem, Z16
|
||||
|
||||
ROUND1(a,b,c,d, 1,0x00, 7)
|
||||
VMOVAPD mem, Z17
|
||||
ROUND1(d,a,b,c, 2,0x01,12)
|
||||
VMOVAPD mem, Z18
|
||||
ROUND1(c,d,a,b, 3,0x02,17)
|
||||
VMOVAPD mem, Z19
|
||||
ROUND1(b,c,d,a, 4,0x03,22)
|
||||
VMOVAPD mem, Z20
|
||||
ROUND1(a,b,c,d, 5,0x04, 7)
|
||||
VMOVAPD mem, Z21
|
||||
ROUND1(d,a,b,c, 6,0x05,12)
|
||||
VMOVAPD mem, Z22
|
||||
ROUND1(c,d,a,b, 7,0x06,17)
|
||||
VMOVAPD mem, Z23
|
||||
ROUND1(b,c,d,a, 8,0x07,22)
|
||||
VMOVAPD mem, Z24
|
||||
ROUND1(a,b,c,d, 9,0x08, 7)
|
||||
VMOVAPD mem, Z25
|
||||
ROUND1(d,a,b,c,10,0x09,12)
|
||||
VMOVAPD mem, Z26
|
||||
ROUND1(c,d,a,b,11,0x0a,17)
|
||||
VMOVAPD mem, Z27
|
||||
ROUND1(b,c,d,a,12,0x0b,22)
|
||||
VMOVAPD mem, Z28
|
||||
ROUND1(a,b,c,d,13,0x0c, 7)
|
||||
VMOVAPD mem, Z29
|
||||
ROUND1(d,a,b,c,14,0x0d,12)
|
||||
VMOVAPD mem, Z30
|
||||
ROUND1(c,d,a,b,15,0x0e,17)
|
||||
VMOVAPD mem, Z31
|
||||
|
||||
ROUND1noload(b,c,d,a, 0x0f,22)
|
||||
|
||||
VMOVAPD d, tmp
|
||||
VMOVAPD d, tmp2
|
||||
|
||||
ROUND2(a,b,c,d, Z17,0x10, 5)
|
||||
ROUND2(d,a,b,c, Z22,0x11, 9)
|
||||
ROUND2(c,d,a,b, Z27,0x12,14)
|
||||
ROUND2(b,c,d,a, Z16,0x13,20)
|
||||
ROUND2(a,b,c,d, Z21,0x14, 5)
|
||||
ROUND2(d,a,b,c, Z26,0x15, 9)
|
||||
ROUND2(c,d,a,b, Z31,0x16,14)
|
||||
ROUND2(b,c,d,a, Z20,0x17,20)
|
||||
ROUND2(a,b,c,d, Z25,0x18, 5)
|
||||
ROUND2(d,a,b,c, Z30,0x19, 9)
|
||||
ROUND2(c,d,a,b, Z19,0x1a,14)
|
||||
ROUND2(b,c,d,a, Z24,0x1b,20)
|
||||
ROUND2(a,b,c,d, Z29,0x1c, 5)
|
||||
ROUND2(d,a,b,c, Z18,0x1d, 9)
|
||||
ROUND2(c,d,a,b, Z23,0x1e,14)
|
||||
ROUND2(b,c,d,a, Z28,0x1f,20)
|
||||
|
||||
VMOVAPD c, tmp
|
||||
|
||||
ROUND3(a,b,c,d, Z21,0x20, 4)
|
||||
ROUND3(d,a,b,c, Z24,0x21,11)
|
||||
ROUND3(c,d,a,b, Z27,0x22,16)
|
||||
ROUND3(b,c,d,a, Z30,0x23,23)
|
||||
ROUND3(a,b,c,d, Z17,0x24, 4)
|
||||
ROUND3(d,a,b,c, Z20,0x25,11)
|
||||
ROUND3(c,d,a,b, Z23,0x26,16)
|
||||
ROUND3(b,c,d,a, Z26,0x27,23)
|
||||
ROUND3(a,b,c,d, Z29,0x28, 4)
|
||||
ROUND3(d,a,b,c, Z16,0x29,11)
|
||||
ROUND3(c,d,a,b, Z19,0x2a,16)
|
||||
ROUND3(b,c,d,a, Z22,0x2b,23)
|
||||
ROUND3(a,b,c,d, Z25,0x2c, 4)
|
||||
ROUND3(d,a,b,c, Z28,0x2d,11)
|
||||
ROUND3(c,d,a,b, Z31,0x2e,16)
|
||||
ROUND3(b,c,d,a, Z18,0x2f,23)
|
||||
|
||||
VPXORQ d, ones, tmp
|
||||
|
||||
ROUND4(a,b,c,d, Z16,0x30, 6)
|
||||
ROUND4(d,a,b,c, Z23,0x31,10)
|
||||
ROUND4(c,d,a,b, Z30,0x32,15)
|
||||
ROUND4(b,c,d,a, Z21,0x33,21)
|
||||
ROUND4(a,b,c,d, Z28,0x34, 6)
|
||||
ROUND4(d,a,b,c, Z19,0x35,10)
|
||||
ROUND4(c,d,a,b, Z26,0x36,15)
|
||||
ROUND4(b,c,d,a, Z17,0x37,21)
|
||||
ROUND4(a,b,c,d, Z24,0x38, 6)
|
||||
ROUND4(d,a,b,c, Z31,0x39,10)
|
||||
ROUND4(c,d,a,b, Z22,0x3a,15)
|
||||
ROUND4(b,c,d,a, Z29,0x3b,21)
|
||||
ROUND4(a,b,c,d, Z20,0x3c, 6)
|
||||
ROUND4(d,a,b,c, Z27,0x3d,10)
|
||||
ROUND4(c,d,a,b, Z18,0x3e,15)
|
||||
ROUND4(b,c,d,a, Z25,0x3f,21)
|
||||
|
||||
VPADDD sa, a, a
|
||||
VPADDD sb, b, b
|
||||
VPADDD sc, c, c
|
||||
VPADDD sd, d, d
|
||||
|
||||
LEAQ 64(base), base
|
||||
SUBQ $64, count
|
||||
JNE loop
|
||||
|
||||
VMOVUPD a, (dig)
|
||||
VMOVUPD b, 0x40(dig)
|
||||
VMOVUPD c, 0x80(dig)
|
||||
VMOVUPD d, 0xc0(dig)
|
||||
|
||||
VZEROUPPER
|
||||
RET
|
||||
+281
@@ -0,0 +1,281 @@
|
||||
//+build !noasm,!appengine,gc
|
||||
|
||||
// Copyright (c) 2018 Igneous Systems
|
||||
// MIT License
|
||||
//
|
||||
// Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
// of this software and associated documentation files (the "Software"), to deal
|
||||
// in the Software without restriction, including without limitation the rights
|
||||
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
// copies of the Software, and to permit persons to whom the Software is
|
||||
// furnished to do so, subject to the following conditions:
|
||||
//
|
||||
// The above copyright notice and this permission notice shall be included in all
|
||||
// copies or substantial portions of the Software.
|
||||
//
|
||||
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
// SOFTWARE.
|
||||
|
||||
// Copyright (c) 2020 MinIO Inc. All rights reserved.
|
||||
// Use of this source code is governed by a license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
// This is the AVX2 implementation of the MD5 block function (8-way parallel)
|
||||
|
||||
// block8(state *uint64, base uintptr, bufs *int32, cache *byte, n int)
|
||||
TEXT ·block8(SB), 4, $0-40
|
||||
MOVQ state+0(FP), BX
|
||||
MOVQ base+8(FP), SI
|
||||
MOVQ bufs+16(FP), AX
|
||||
MOVQ cache+24(FP), CX
|
||||
MOVQ n+32(FP), DX
|
||||
MOVQ ·avx256md5consts+0(SB), DI
|
||||
|
||||
// Align cache (which is stack allocated by the compiler)
|
||||
// to a 256 bit boundary (ymm register alignment)
|
||||
// The cache8 type is deliberately oversized to permit this.
|
||||
ADDQ $31, CX
|
||||
ANDB $-32, CL
|
||||
|
||||
#define a Y0
|
||||
#define b Y1
|
||||
#define c Y2
|
||||
#define d Y3
|
||||
|
||||
#define sa Y4
|
||||
#define sb Y5
|
||||
#define sc Y6
|
||||
#define sd Y7
|
||||
|
||||
#define tmp Y8
|
||||
#define tmp2 Y9
|
||||
|
||||
#define mask Y10
|
||||
#define off Y11
|
||||
|
||||
#define ones Y12
|
||||
|
||||
#define rtmp1 Y13
|
||||
#define rtmp2 Y14
|
||||
|
||||
#define mem Y15
|
||||
|
||||
#define dig BX
|
||||
#define cache CX
|
||||
#define count DX
|
||||
#define base SI
|
||||
#define consts DI
|
||||
|
||||
#define prepmask \
|
||||
VPXOR mask, mask, mask \
|
||||
VPCMPGTD mask, off, mask
|
||||
|
||||
#define prep(index) \
|
||||
VMOVAPD mask, rtmp2 \
|
||||
VPGATHERDD rtmp2, index*4(base)(off*1), mem
|
||||
|
||||
#define load(index) \
|
||||
VMOVAPD index*32(cache), mem
|
||||
|
||||
#define store(index) \
|
||||
VMOVAPD mem, index*32(cache)
|
||||
|
||||
#define roll(shift, a) \
|
||||
VPSLLD $shift, a, rtmp1 \
|
||||
VPSRLD $32-shift, a, a \
|
||||
VPOR rtmp1, a, a
|
||||
|
||||
#define ROUND1(a, b, c, d, index, const, shift) \
|
||||
VPXOR c, tmp, tmp \
|
||||
VPADDD 32*const(consts), a, a \
|
||||
VPADDD mem, a, a \
|
||||
VPAND b, tmp, tmp \
|
||||
VPXOR d, tmp, tmp \
|
||||
prep(index) \
|
||||
VPADDD tmp, a, a \
|
||||
roll(shift,a) \
|
||||
VMOVAPD c, tmp \
|
||||
VPADDD b, a, a
|
||||
|
||||
#define ROUND1load(a, b, c, d, index, const, shift) \
|
||||
VXORPD c, tmp, tmp \
|
||||
VPADDD 32*const(consts), a, a \
|
||||
VPADDD mem, a, a \
|
||||
VPAND b, tmp, tmp \
|
||||
VPXOR d, tmp, tmp \
|
||||
load(index) \
|
||||
VPADDD tmp, a, a \
|
||||
roll(shift,a) \
|
||||
VMOVAPD c, tmp \
|
||||
VPADDD b, a, a
|
||||
|
||||
#define ROUND2(a, b, c, d, index, const, shift) \
|
||||
VPADDD 32*const(consts), a, a \
|
||||
VPADDD mem, a, a \
|
||||
VPAND b, tmp2, tmp2 \
|
||||
VANDNPD c, tmp, tmp \
|
||||
load(index) \
|
||||
VPOR tmp, tmp2, tmp2 \
|
||||
VMOVAPD c, tmp \
|
||||
VPADDD tmp2, a, a \
|
||||
VMOVAPD c, tmp2 \
|
||||
roll(shift,a) \
|
||||
VPADDD b, a, a
|
||||
|
||||
#define ROUND3(a, b, c, d, index, const, shift) \
|
||||
VPADDD 32*const(consts), a, a \
|
||||
VPADDD mem, a, a \
|
||||
load(index) \
|
||||
VPXOR d, tmp, tmp \
|
||||
VPXOR b, tmp, tmp \
|
||||
VPADDD tmp, a, a \
|
||||
roll(shift,a) \
|
||||
VMOVAPD b, tmp \
|
||||
VPADDD b, a, a
|
||||
|
||||
#define ROUND4(a, b, c, d, index, const, shift) \
|
||||
VPADDD 32*const(consts), a, a \
|
||||
VPADDD mem, a, a \
|
||||
VPOR b, tmp, tmp \
|
||||
VPXOR c, tmp, tmp \
|
||||
VPADDD tmp, a, a \
|
||||
load(index) \
|
||||
roll(shift,a) \
|
||||
VPXOR c, ones, tmp \
|
||||
VPADDD b, a, a
|
||||
|
||||
// load digest into state registers
|
||||
VMOVUPD (dig), a
|
||||
VMOVUPD 32(dig), b
|
||||
VMOVUPD 64(dig), c
|
||||
VMOVUPD 96(dig), d
|
||||
|
||||
// load source buffer offsets
|
||||
VMOVUPD (AX), off
|
||||
|
||||
prepmask
|
||||
VPCMPEQD ones, ones, ones
|
||||
|
||||
loop:
|
||||
VMOVAPD a, sa
|
||||
VMOVAPD b, sb
|
||||
VMOVAPD c, sc
|
||||
VMOVAPD d, sd
|
||||
|
||||
prep(0)
|
||||
VMOVAPD d, tmp
|
||||
store(0)
|
||||
|
||||
ROUND1(a,b,c,d, 1,0x00, 7)
|
||||
store(1)
|
||||
ROUND1(d,a,b,c, 2,0x01,12)
|
||||
store(2)
|
||||
ROUND1(c,d,a,b, 3,0x02,17)
|
||||
store(3)
|
||||
ROUND1(b,c,d,a, 4,0x03,22)
|
||||
store(4)
|
||||
ROUND1(a,b,c,d, 5,0x04, 7)
|
||||
store(5)
|
||||
ROUND1(d,a,b,c, 6,0x05,12)
|
||||
store(6)
|
||||
ROUND1(c,d,a,b, 7,0x06,17)
|
||||
store(7)
|
||||
ROUND1(b,c,d,a, 8,0x07,22)
|
||||
store(8)
|
||||
ROUND1(a,b,c,d, 9,0x08, 7)
|
||||
store(9)
|
||||
ROUND1(d,a,b,c,10,0x09,12)
|
||||
store(10)
|
||||
ROUND1(c,d,a,b,11,0x0a,17)
|
||||
store(11)
|
||||
ROUND1(b,c,d,a,12,0x0b,22)
|
||||
store(12)
|
||||
ROUND1(a,b,c,d,13,0x0c, 7)
|
||||
store(13)
|
||||
ROUND1(d,a,b,c,14,0x0d,12)
|
||||
store(14)
|
||||
ROUND1(c,d,a,b,15,0x0e,17)
|
||||
store(15)
|
||||
ROUND1load(b,c,d,a, 1,0x0f,22)
|
||||
|
||||
VMOVAPD d, tmp
|
||||
VMOVAPD d, tmp2
|
||||
|
||||
ROUND2(a,b,c,d, 6,0x10, 5)
|
||||
ROUND2(d,a,b,c,11,0x11, 9)
|
||||
ROUND2(c,d,a,b, 0,0x12,14)
|
||||
ROUND2(b,c,d,a, 5,0x13,20)
|
||||
ROUND2(a,b,c,d,10,0x14, 5)
|
||||
ROUND2(d,a,b,c,15,0x15, 9)
|
||||
ROUND2(c,d,a,b, 4,0x16,14)
|
||||
ROUND2(b,c,d,a, 9,0x17,20)
|
||||
ROUND2(a,b,c,d,14,0x18, 5)
|
||||
ROUND2(d,a,b,c, 3,0x19, 9)
|
||||
ROUND2(c,d,a,b, 8,0x1a,14)
|
||||
ROUND2(b,c,d,a,13,0x1b,20)
|
||||
ROUND2(a,b,c,d, 2,0x1c, 5)
|
||||
ROUND2(d,a,b,c, 7,0x1d, 9)
|
||||
ROUND2(c,d,a,b,12,0x1e,14)
|
||||
ROUND2(b,c,d,a, 0,0x1f,20)
|
||||
|
||||
load(5)
|
||||
VMOVAPD c, tmp
|
||||
|
||||
ROUND3(a,b,c,d, 8,0x20, 4)
|
||||
ROUND3(d,a,b,c,11,0x21,11)
|
||||
ROUND3(c,d,a,b,14,0x22,16)
|
||||
ROUND3(b,c,d,a, 1,0x23,23)
|
||||
ROUND3(a,b,c,d, 4,0x24, 4)
|
||||
ROUND3(d,a,b,c, 7,0x25,11)
|
||||
ROUND3(c,d,a,b,10,0x26,16)
|
||||
ROUND3(b,c,d,a,13,0x27,23)
|
||||
ROUND3(a,b,c,d, 0,0x28, 4)
|
||||
ROUND3(d,a,b,c, 3,0x29,11)
|
||||
ROUND3(c,d,a,b, 6,0x2a,16)
|
||||
ROUND3(b,c,d,a, 9,0x2b,23)
|
||||
ROUND3(a,b,c,d,12,0x2c, 4)
|
||||
ROUND3(d,a,b,c,15,0x2d,11)
|
||||
ROUND3(c,d,a,b, 2,0x2e,16)
|
||||
ROUND3(b,c,d,a, 0,0x2f,23)
|
||||
|
||||
load(0)
|
||||
VPXOR d, ones, tmp
|
||||
|
||||
ROUND4(a,b,c,d, 7,0x30, 6)
|
||||
ROUND4(d,a,b,c,14,0x31,10)
|
||||
ROUND4(c,d,a,b, 5,0x32,15)
|
||||
ROUND4(b,c,d,a,12,0x33,21)
|
||||
ROUND4(a,b,c,d, 3,0x34, 6)
|
||||
ROUND4(d,a,b,c,10,0x35,10)
|
||||
ROUND4(c,d,a,b, 1,0x36,15)
|
||||
ROUND4(b,c,d,a, 8,0x37,21)
|
||||
ROUND4(a,b,c,d,15,0x38, 6)
|
||||
ROUND4(d,a,b,c, 6,0x39,10)
|
||||
ROUND4(c,d,a,b,13,0x3a,15)
|
||||
ROUND4(b,c,d,a, 4,0x3b,21)
|
||||
ROUND4(a,b,c,d,11,0x3c, 6)
|
||||
ROUND4(d,a,b,c, 2,0x3d,10)
|
||||
ROUND4(c,d,a,b, 9,0x3e,15)
|
||||
ROUND4(b,c,d,a, 0,0x3f,21)
|
||||
|
||||
VPADDD sa, a, a
|
||||
VPADDD sb, b, b
|
||||
VPADDD sc, c, c
|
||||
VPADDD sd, d, d
|
||||
|
||||
LEAQ 64(base), base
|
||||
SUBQ $64, count
|
||||
JNE loop
|
||||
|
||||
VMOVUPD a, (dig)
|
||||
VMOVUPD b, 32(dig)
|
||||
VMOVUPD c, 64(dig)
|
||||
VMOVUPD d, 96(dig)
|
||||
|
||||
VZEROUPPER
|
||||
RET
|
||||
+210
@@ -0,0 +1,210 @@
|
||||
//+build !noasm,!appengine,gc
|
||||
|
||||
// Copyright (c) 2020 MinIO Inc. All rights reserved.
|
||||
// Use of this source code is governed by a license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
package md5simd
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"math"
|
||||
"unsafe"
|
||||
|
||||
"github.com/klauspost/cpuid/v2"
|
||||
)
|
||||
|
||||
var hasAVX512 bool
|
||||
|
||||
func init() {
|
||||
// VANDNPD requires AVX512DQ. Technically it could be VPTERNLOGQ which is AVX512F.
|
||||
hasAVX512 = cpuid.CPU.Supports(cpuid.AVX512F, cpuid.AVX512DQ)
|
||||
}
|
||||
|
||||
//go:noescape
|
||||
func block8(state *uint32, base uintptr, bufs *int32, cache *byte, n int)
|
||||
|
||||
//go:noescape
|
||||
func block16(state *uint32, base uintptr, ptrs *int32, mask uint64, n int)
|
||||
|
||||
// 8-way 4x uint32 digests in 4 ymm registers
|
||||
// (ymm0, ymm1, ymm2, ymm3)
|
||||
type digest8 struct {
|
||||
v0, v1, v2, v3 [8]uint32
|
||||
}
|
||||
|
||||
// Stack cache for 8x64 byte md5.BlockSize bytes.
|
||||
// Must be 32-byte aligned, so allocate 512+32 and
|
||||
// align upwards at runtime.
|
||||
type cache8 [512 + 32]byte
|
||||
|
||||
// MD5 magic numbers for one lane of hashing; inflated
|
||||
// 8x below at init time.
|
||||
var md5consts = [64]uint32{
|
||||
0xd76aa478, 0xe8c7b756, 0x242070db, 0xc1bdceee,
|
||||
0xf57c0faf, 0x4787c62a, 0xa8304613, 0xfd469501,
|
||||
0x698098d8, 0x8b44f7af, 0xffff5bb1, 0x895cd7be,
|
||||
0x6b901122, 0xfd987193, 0xa679438e, 0x49b40821,
|
||||
0xf61e2562, 0xc040b340, 0x265e5a51, 0xe9b6c7aa,
|
||||
0xd62f105d, 0x02441453, 0xd8a1e681, 0xe7d3fbc8,
|
||||
0x21e1cde6, 0xc33707d6, 0xf4d50d87, 0x455a14ed,
|
||||
0xa9e3e905, 0xfcefa3f8, 0x676f02d9, 0x8d2a4c8a,
|
||||
0xfffa3942, 0x8771f681, 0x6d9d6122, 0xfde5380c,
|
||||
0xa4beea44, 0x4bdecfa9, 0xf6bb4b60, 0xbebfbc70,
|
||||
0x289b7ec6, 0xeaa127fa, 0xd4ef3085, 0x04881d05,
|
||||
0xd9d4d039, 0xe6db99e5, 0x1fa27cf8, 0xc4ac5665,
|
||||
0xf4292244, 0x432aff97, 0xab9423a7, 0xfc93a039,
|
||||
0x655b59c3, 0x8f0ccc92, 0xffeff47d, 0x85845dd1,
|
||||
0x6fa87e4f, 0xfe2ce6e0, 0xa3014314, 0x4e0811a1,
|
||||
0xf7537e82, 0xbd3af235, 0x2ad7d2bb, 0xeb86d391,
|
||||
}
|
||||
|
||||
// inflate the consts 8-way for 8x md5 (256 bit ymm registers)
|
||||
var avx256md5consts = func(c []uint32) []uint32 {
|
||||
inf := make([]uint32, 8*len(c))
|
||||
for i := range c {
|
||||
for j := 0; j < 8; j++ {
|
||||
inf[(i*8)+j] = c[i]
|
||||
}
|
||||
}
|
||||
return inf
|
||||
}(md5consts[:])
|
||||
|
||||
// 16-way 4x uint32 digests in 4 zmm registers
|
||||
type digest16 struct {
|
||||
v0, v1, v2, v3 [16]uint32
|
||||
}
|
||||
|
||||
// inflate the consts 16-way for 16x md5 (512 bit zmm registers)
|
||||
var avx512md5consts = func(c []uint32) []uint32 {
|
||||
inf := make([]uint32, 16*len(c))
|
||||
for i := range c {
|
||||
for j := 0; j < 16; j++ {
|
||||
inf[(i*16)+j] = c[i]
|
||||
}
|
||||
}
|
||||
return inf
|
||||
}(md5consts[:])
|
||||
|
||||
// Interface function to assembly code
|
||||
func (s *md5Server) blockMd5_x16(d *digest16, input [16][]byte, half bool) {
|
||||
if hasAVX512 {
|
||||
blockMd5_avx512(d, input, s.allBufs, &s.maskRounds16)
|
||||
return
|
||||
}
|
||||
|
||||
// Preparing data using copy is slower since copies aren't inlined.
|
||||
|
||||
// Calculate on this goroutine
|
||||
if half {
|
||||
for i := range s.i8[0][:] {
|
||||
s.i8[0][i] = input[i]
|
||||
}
|
||||
for i := range s.d8a.v0[:] {
|
||||
s.d8a.v0[i], s.d8a.v1[i], s.d8a.v2[i], s.d8a.v3[i] = d.v0[i], d.v1[i], d.v2[i], d.v3[i]
|
||||
}
|
||||
blockMd5_avx2(&s.d8a, s.i8[0], s.allBufs, &s.maskRounds8a)
|
||||
for i := range s.d8a.v0[:] {
|
||||
d.v0[i], d.v1[i], d.v2[i], d.v3[i] = s.d8a.v0[i], s.d8a.v1[i], s.d8a.v2[i], s.d8a.v3[i]
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
for i := range s.i8[0][:] {
|
||||
s.i8[0][i], s.i8[1][i] = input[i], input[8+i]
|
||||
}
|
||||
|
||||
for i := range s.d8a.v0[:] {
|
||||
j := (i + 8) & 15
|
||||
s.d8a.v0[i], s.d8a.v1[i], s.d8a.v2[i], s.d8a.v3[i] = d.v0[i], d.v1[i], d.v2[i], d.v3[i]
|
||||
s.d8b.v0[i], s.d8b.v1[i], s.d8b.v2[i], s.d8b.v3[i] = d.v0[j], d.v1[j], d.v2[j], d.v3[j]
|
||||
}
|
||||
|
||||
// Benchmarks appears to be slightly faster when spinning up 2 goroutines instead
|
||||
// of using the current for one of the blocks.
|
||||
s.wg.Add(2)
|
||||
go func() { blockMd5_avx2(&s.d8a, s.i8[0], s.allBufs, &s.maskRounds8a); s.wg.Done() }()
|
||||
go func() { blockMd5_avx2(&s.d8b, s.i8[1], s.allBufs, &s.maskRounds8b); s.wg.Done() }()
|
||||
s.wg.Wait()
|
||||
for i := range s.d8a.v0[:] {
|
||||
d.v0[i], d.v1[i], d.v2[i], d.v3[i] = s.d8a.v0[i], s.d8a.v1[i], s.d8a.v2[i], s.d8a.v3[i]
|
||||
}
|
||||
for i := range s.d8b.v0[:] {
|
||||
j := (i + 8) & 15
|
||||
d.v0[j], d.v1[j], d.v2[j], d.v3[j] = s.d8b.v0[i], s.d8b.v1[i], s.d8b.v2[i], s.d8b.v3[i]
|
||||
}
|
||||
}
|
||||
|
||||
// Interface function to AVX512 assembly code
|
||||
func blockMd5_avx512(s *digest16, input [16][]byte, base []byte, maskRounds *[16]maskRounds) {
|
||||
baseMin := uint64(uintptr(unsafe.Pointer(&(base[0]))))
|
||||
ptrs := [16]int32{}
|
||||
|
||||
for i := range ptrs {
|
||||
if len(input[i]) > 0 {
|
||||
if len(input[i]) > internalBlockSize {
|
||||
panic(fmt.Sprintf("Sanity check fails for lane %d: maximum input length cannot exceed internalBlockSize", i))
|
||||
}
|
||||
|
||||
off := uint64(uintptr(unsafe.Pointer(&(input[i][0])))) - baseMin
|
||||
if off > math.MaxUint32 {
|
||||
panic(fmt.Sprintf("invalid buffer sent with offset %x", off))
|
||||
}
|
||||
ptrs[i] = int32(off)
|
||||
}
|
||||
}
|
||||
|
||||
sdup := *s // create copy of initial states to receive intermediate updates
|
||||
|
||||
rounds := generateMaskAndRounds16(input, maskRounds)
|
||||
|
||||
for r := 0; r < rounds; r++ {
|
||||
m := maskRounds[r]
|
||||
|
||||
block16(&sdup.v0[0], uintptr(baseMin), &ptrs[0], m.mask, int(64*m.rounds))
|
||||
|
||||
for j := 0; j < len(ptrs); j++ {
|
||||
ptrs[j] += int32(64 * m.rounds) // update pointers for next round
|
||||
if m.mask&(1<<j) != 0 { // update digest if still masked as active
|
||||
(*s).v0[j], (*s).v1[j], (*s).v2[j], (*s).v3[j] = sdup.v0[j], sdup.v1[j], sdup.v2[j], sdup.v3[j]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Interface function to AVX2 assembly code
|
||||
func blockMd5_avx2(s *digest8, input [8][]byte, base []byte, maskRounds *[8]maskRounds) {
|
||||
baseMin := uint64(uintptr(unsafe.Pointer(&(base[0])))) - 4
|
||||
ptrs := [8]int32{}
|
||||
|
||||
for i := range ptrs {
|
||||
if len(input[i]) > 0 {
|
||||
if len(input[i]) > internalBlockSize {
|
||||
panic(fmt.Sprintf("Sanity check fails for lane %d: maximum input length cannot exceed internalBlockSize", i))
|
||||
}
|
||||
|
||||
off := uint64(uintptr(unsafe.Pointer(&(input[i][0])))) - baseMin
|
||||
if off > math.MaxUint32 {
|
||||
panic(fmt.Sprintf("invalid buffer sent with offset %x", off))
|
||||
}
|
||||
ptrs[i] = int32(off)
|
||||
}
|
||||
}
|
||||
|
||||
sdup := *s // create copy of initial states to receive intermediate updates
|
||||
|
||||
rounds := generateMaskAndRounds8(input, maskRounds)
|
||||
|
||||
for r := 0; r < rounds; r++ {
|
||||
m := maskRounds[r]
|
||||
var cache cache8 // stack storage for block8 tmp state
|
||||
block8(&sdup.v0[0], uintptr(baseMin), &ptrs[0], &cache[0], int(64*m.rounds))
|
||||
|
||||
for j := 0; j < len(ptrs); j++ {
|
||||
ptrs[j] += int32(64 * m.rounds) // update pointers for next round
|
||||
if m.mask&(1<<j) != 0 { // update digest if still masked as active
|
||||
(*s).v0[j], (*s).v1[j], (*s).v2[j], (*s).v3[j] = sdup.v0[j], sdup.v1[j], sdup.v2[j], sdup.v3[j]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
+188
@@ -0,0 +1,188 @@
|
||||
//+build !noasm,!appengine,gc
|
||||
|
||||
// Copyright (c) 2020 MinIO Inc. All rights reserved.
|
||||
// Use of this source code is governed by a license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
package md5simd
|
||||
|
||||
import (
|
||||
"encoding/binary"
|
||||
"errors"
|
||||
"fmt"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
)
|
||||
|
||||
// md5Digest - Type for computing MD5 using either AVX2 or AVX512
|
||||
type md5Digest struct {
|
||||
uid uint64
|
||||
blocksCh chan blockInput
|
||||
cycleServer chan uint64
|
||||
x [BlockSize]byte
|
||||
nx int
|
||||
len uint64
|
||||
buffers <-chan []byte
|
||||
}
|
||||
|
||||
// NewHash - initialize instance for Md5 implementation.
|
||||
func (s *md5Server) NewHash() Hasher {
|
||||
uid := atomic.AddUint64(&s.uidCounter, 1)
|
||||
blockCh := make(chan blockInput, buffersPerLane)
|
||||
s.newInput <- newClient{
|
||||
uid: uid,
|
||||
input: blockCh,
|
||||
}
|
||||
return &md5Digest{
|
||||
uid: uid,
|
||||
buffers: s.buffers,
|
||||
blocksCh: blockCh,
|
||||
cycleServer: s.cycle,
|
||||
}
|
||||
}
|
||||
|
||||
// Size - Return size of checksum
|
||||
func (d *md5Digest) Size() int { return Size }
|
||||
|
||||
// BlockSize - Return blocksize of checksum
|
||||
func (d md5Digest) BlockSize() int { return BlockSize }
|
||||
|
||||
func (d *md5Digest) Reset() {
|
||||
if d.blocksCh == nil {
|
||||
panic("reset after close")
|
||||
}
|
||||
d.nx = 0
|
||||
d.len = 0
|
||||
d.sendBlock(blockInput{uid: d.uid, reset: true}, false)
|
||||
}
|
||||
|
||||
// write to digest
|
||||
func (d *md5Digest) Write(p []byte) (nn int, err error) {
|
||||
if d.blocksCh == nil {
|
||||
return 0, errors.New("md5Digest closed")
|
||||
}
|
||||
|
||||
// break input into chunks of maximum internalBlockSize size
|
||||
for {
|
||||
l := len(p)
|
||||
if l > internalBlockSize {
|
||||
l = internalBlockSize
|
||||
}
|
||||
nnn, err := d.write(p[:l])
|
||||
if err != nil {
|
||||
return nn, err
|
||||
}
|
||||
nn += nnn
|
||||
p = p[l:]
|
||||
|
||||
if len(p) == 0 {
|
||||
break
|
||||
}
|
||||
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
func (d *md5Digest) write(p []byte) (nn int, err error) {
|
||||
|
||||
nn = len(p)
|
||||
d.len += uint64(nn)
|
||||
if d.nx > 0 {
|
||||
n := copy(d.x[d.nx:], p)
|
||||
d.nx += n
|
||||
if d.nx == BlockSize {
|
||||
// Create a copy of the overflow buffer in order to send it async over the channel
|
||||
// (since we will modify the overflow buffer down below with any access beyond multiples of 64)
|
||||
tmp := <-d.buffers
|
||||
tmp = tmp[:BlockSize]
|
||||
copy(tmp, d.x[:])
|
||||
d.sendBlock(blockInput{uid: d.uid, msg: tmp}, len(p)-n < BlockSize)
|
||||
d.nx = 0
|
||||
}
|
||||
p = p[n:]
|
||||
}
|
||||
if len(p) >= BlockSize {
|
||||
n := len(p) &^ (BlockSize - 1)
|
||||
buf := <-d.buffers
|
||||
buf = buf[:n]
|
||||
copy(buf, p)
|
||||
d.sendBlock(blockInput{uid: d.uid, msg: buf}, len(p)-n < BlockSize)
|
||||
p = p[n:]
|
||||
}
|
||||
if len(p) > 0 {
|
||||
d.nx = copy(d.x[:], p)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
func (d *md5Digest) Close() {
|
||||
if d.blocksCh != nil {
|
||||
close(d.blocksCh)
|
||||
d.blocksCh = nil
|
||||
}
|
||||
}
|
||||
|
||||
var sumChPool sync.Pool
|
||||
|
||||
func init() {
|
||||
sumChPool.New = func() interface{} {
|
||||
return make(chan sumResult, 1)
|
||||
}
|
||||
}
|
||||
|
||||
// Sum - Return MD5 sum in bytes
|
||||
func (d *md5Digest) Sum(in []byte) (result []byte) {
|
||||
if d.blocksCh == nil {
|
||||
panic("sum after close")
|
||||
}
|
||||
|
||||
trail := <-d.buffers
|
||||
trail = append(trail[:0], d.x[:d.nx]...)
|
||||
|
||||
length := d.len
|
||||
// Padding. Add a 1 bit and 0 bits until 56 bytes mod 64.
|
||||
var tmp [64]byte
|
||||
tmp[0] = 0x80
|
||||
if length%64 < 56 {
|
||||
trail = append(trail, tmp[0:56-length%64]...)
|
||||
} else {
|
||||
trail = append(trail, tmp[0:64+56-length%64]...)
|
||||
}
|
||||
|
||||
// Length in bits.
|
||||
length <<= 3
|
||||
binary.LittleEndian.PutUint64(tmp[:], length) // append length in bits
|
||||
|
||||
trail = append(trail, tmp[0:8]...)
|
||||
if len(trail)%BlockSize != 0 {
|
||||
panic(fmt.Errorf("internal error: sum block was not aligned. len=%d, nx=%d", len(trail), d.nx))
|
||||
}
|
||||
sumCh := sumChPool.Get().(chan sumResult)
|
||||
d.sendBlock(blockInput{uid: d.uid, msg: trail, sumCh: sumCh}, true)
|
||||
|
||||
sum := <-sumCh
|
||||
sumChPool.Put(sumCh)
|
||||
|
||||
return append(in, sum.digest[:]...)
|
||||
}
|
||||
|
||||
// sendBlock will send a block for processing.
|
||||
// If cycle is true we will block on cycle, otherwise we will only block
|
||||
// if the block channel is full.
|
||||
func (d *md5Digest) sendBlock(bi blockInput, cycle bool) {
|
||||
if cycle {
|
||||
select {
|
||||
case d.blocksCh <- bi:
|
||||
d.cycleServer <- d.uid
|
||||
}
|
||||
return
|
||||
}
|
||||
// Only block on cycle if we filled the buffer
|
||||
select {
|
||||
case d.blocksCh <- bi:
|
||||
return
|
||||
default:
|
||||
d.cycleServer <- d.uid
|
||||
d.blocksCh <- bi
|
||||
}
|
||||
}
|
||||
+397
@@ -0,0 +1,397 @@
|
||||
//+build !noasm,!appengine,gc
|
||||
|
||||
// Copyright (c) 2020 MinIO Inc. All rights reserved.
|
||||
// Use of this source code is governed by a license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
package md5simd
|
||||
|
||||
import (
|
||||
"encoding/binary"
|
||||
"fmt"
|
||||
"runtime"
|
||||
"sync"
|
||||
|
||||
"github.com/klauspost/cpuid/v2"
|
||||
)
|
||||
|
||||
// MD5 initialization constants
|
||||
const (
|
||||
// Lanes is the number of concurrently calculated hashes.
|
||||
Lanes = 16
|
||||
|
||||
init0 = 0x67452301
|
||||
init1 = 0xefcdab89
|
||||
init2 = 0x98badcfe
|
||||
init3 = 0x10325476
|
||||
|
||||
// Use scalar routine when below this many lanes
|
||||
useScalarBelow = 3
|
||||
)
|
||||
|
||||
// md5ServerUID - Does not start at 0 but next multiple of 16 so as to be able to
|
||||
// differentiate with default initialisation value of 0
|
||||
const md5ServerUID = Lanes
|
||||
|
||||
const buffersPerLane = 3
|
||||
|
||||
// Message to send across input channel
|
||||
type blockInput struct {
|
||||
uid uint64
|
||||
msg []byte
|
||||
sumCh chan sumResult
|
||||
reset bool
|
||||
}
|
||||
|
||||
type sumResult struct {
|
||||
digest [Size]byte
|
||||
}
|
||||
|
||||
type lanesInfo [Lanes]blockInput
|
||||
|
||||
// md5Server - Type to implement parallel handling of MD5 invocations
|
||||
type md5Server struct {
|
||||
uidCounter uint64
|
||||
cycle chan uint64 // client with uid has update.
|
||||
newInput chan newClient // Add new client.
|
||||
digests map[uint64][Size]byte // Map of uids to (interim) digest results
|
||||
maskRounds16 [16]maskRounds // Pre-allocated static array for max 16 rounds
|
||||
maskRounds8a [8]maskRounds // Pre-allocated static array for max 8 rounds (1st AVX2 core)
|
||||
maskRounds8b [8]maskRounds // Pre-allocated static array for max 8 rounds (2nd AVX2 core)
|
||||
allBufs []byte // Preallocated buffer.
|
||||
buffers chan []byte // Preallocated buffers, sliced from allBufs.
|
||||
|
||||
i8 [2][8][]byte // avx2 temporary vars
|
||||
d8a, d8b digest8
|
||||
wg sync.WaitGroup
|
||||
}
|
||||
|
||||
// NewServer - Create new object for parallel processing handling
|
||||
func NewServer() Server {
|
||||
if !cpuid.CPU.Supports(cpuid.AVX2) {
|
||||
return &fallbackServer{}
|
||||
}
|
||||
md5srv := &md5Server{}
|
||||
md5srv.digests = make(map[uint64][Size]byte)
|
||||
md5srv.newInput = make(chan newClient, Lanes)
|
||||
md5srv.cycle = make(chan uint64, Lanes*10)
|
||||
md5srv.uidCounter = md5ServerUID - 1
|
||||
md5srv.allBufs = make([]byte, 32+buffersPerLane*Lanes*internalBlockSize)
|
||||
md5srv.buffers = make(chan []byte, buffersPerLane*Lanes)
|
||||
// Fill buffers.
|
||||
for i := 0; i < buffersPerLane*Lanes; i++ {
|
||||
s := 32 + i*internalBlockSize
|
||||
md5srv.buffers <- md5srv.allBufs[s : s+internalBlockSize : s+internalBlockSize]
|
||||
}
|
||||
|
||||
// Start a single thread for reading from the input channel
|
||||
go md5srv.process(md5srv.newInput)
|
||||
return md5srv
|
||||
}
|
||||
|
||||
type newClient struct {
|
||||
uid uint64
|
||||
input chan blockInput
|
||||
}
|
||||
|
||||
// process - Sole handler for reading from the input channel.
|
||||
func (s *md5Server) process(newClients chan newClient) {
|
||||
// To fill up as many lanes as possible:
|
||||
//
|
||||
// 1. Wait for a cycle id.
|
||||
// 2. If not already in a lane, add, otherwise leave on channel
|
||||
// 3. Start timer
|
||||
// 4. Check if lanes is full, if so, goto 10 (process).
|
||||
// 5. If timeout, goto 10.
|
||||
// 6. Wait for new id (goto 2) or timeout (goto 10).
|
||||
// 10. Process.
|
||||
// 11. Check all input if there is already input, if so add to lanes.
|
||||
// 12. Goto 1
|
||||
|
||||
// lanes contains the lanes.
|
||||
var lanes lanesInfo
|
||||
// lanesFilled contains the number of filled lanes for current cycle.
|
||||
var lanesFilled int
|
||||
// clients contains active clients
|
||||
var clients = make(map[uint64]chan blockInput, Lanes)
|
||||
|
||||
addToLane := func(uid uint64) {
|
||||
cl, ok := clients[uid]
|
||||
if !ok {
|
||||
// Unknown client. Maybe it was already removed.
|
||||
return
|
||||
}
|
||||
// Check if we already have it.
|
||||
for _, lane := range lanes[:lanesFilled] {
|
||||
if lane.uid == uid {
|
||||
return
|
||||
}
|
||||
}
|
||||
// Continue until we get a block or there is nothing on channel
|
||||
for {
|
||||
select {
|
||||
case block, ok := <-cl:
|
||||
if !ok {
|
||||
// Client disconnected
|
||||
delete(clients, block.uid)
|
||||
return
|
||||
}
|
||||
if block.uid != uid {
|
||||
panic(fmt.Errorf("uid mismatch, %d (block) != %d (client)", block.uid, uid))
|
||||
}
|
||||
// If reset message, reset and we're done
|
||||
if block.reset {
|
||||
delete(s.digests, uid)
|
||||
continue
|
||||
}
|
||||
|
||||
// If requesting sum, we will need to maintain state.
|
||||
if block.sumCh != nil {
|
||||
var dig digest
|
||||
d, ok := s.digests[uid]
|
||||
if ok {
|
||||
dig.s[0] = binary.LittleEndian.Uint32(d[0:4])
|
||||
dig.s[1] = binary.LittleEndian.Uint32(d[4:8])
|
||||
dig.s[2] = binary.LittleEndian.Uint32(d[8:12])
|
||||
dig.s[3] = binary.LittleEndian.Uint32(d[12:16])
|
||||
} else {
|
||||
dig.s[0], dig.s[1], dig.s[2], dig.s[3] = init0, init1, init2, init3
|
||||
}
|
||||
|
||||
sum := sumResult{}
|
||||
// Add end block to current digest.
|
||||
blockScalar(&dig.s, block.msg)
|
||||
|
||||
binary.LittleEndian.PutUint32(sum.digest[0:], dig.s[0])
|
||||
binary.LittleEndian.PutUint32(sum.digest[4:], dig.s[1])
|
||||
binary.LittleEndian.PutUint32(sum.digest[8:], dig.s[2])
|
||||
binary.LittleEndian.PutUint32(sum.digest[12:], dig.s[3])
|
||||
block.sumCh <- sum
|
||||
if block.msg != nil {
|
||||
s.buffers <- block.msg
|
||||
}
|
||||
continue
|
||||
}
|
||||
if len(block.msg) == 0 {
|
||||
continue
|
||||
}
|
||||
lanes[lanesFilled] = block
|
||||
lanesFilled++
|
||||
return
|
||||
default:
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
addNewClient := func(cl newClient) {
|
||||
if _, ok := clients[cl.uid]; ok {
|
||||
panic("internal error: duplicate client registration")
|
||||
}
|
||||
clients[cl.uid] = cl.input
|
||||
}
|
||||
|
||||
allLanesFilled := func() bool {
|
||||
return lanesFilled == Lanes || lanesFilled >= len(clients)
|
||||
}
|
||||
|
||||
for {
|
||||
// Step 1.
|
||||
for lanesFilled == 0 {
|
||||
select {
|
||||
case cl, ok := <-newClients:
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
addNewClient(cl)
|
||||
// Check if it already sent a payload.
|
||||
addToLane(cl.uid)
|
||||
continue
|
||||
case uid := <-s.cycle:
|
||||
addToLane(uid)
|
||||
}
|
||||
}
|
||||
|
||||
fillLanes:
|
||||
for !allLanesFilled() {
|
||||
select {
|
||||
case cl, ok := <-newClients:
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
addNewClient(cl)
|
||||
|
||||
case uid := <-s.cycle:
|
||||
addToLane(uid)
|
||||
default:
|
||||
// Nothing more queued...
|
||||
break fillLanes
|
||||
}
|
||||
}
|
||||
|
||||
// If we did not fill all lanes, check if there is more waiting
|
||||
if !allLanesFilled() {
|
||||
runtime.Gosched()
|
||||
for uid := range clients {
|
||||
addToLane(uid)
|
||||
if allLanesFilled() {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
if false {
|
||||
if !allLanesFilled() {
|
||||
fmt.Println("Not all lanes filled", lanesFilled, "of", len(clients))
|
||||
//pprof.Lookup("goroutine").WriteTo(os.Stdout, 1)
|
||||
} else if true {
|
||||
fmt.Println("all lanes filled")
|
||||
}
|
||||
}
|
||||
// Process the lanes we could collect
|
||||
s.blocks(lanes[:lanesFilled])
|
||||
|
||||
// Clear lanes...
|
||||
lanesFilled = 0
|
||||
// Add all current queued
|
||||
for uid := range clients {
|
||||
addToLane(uid)
|
||||
if allLanesFilled() {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (s *md5Server) Close() {
|
||||
if s.newInput != nil {
|
||||
close(s.newInput)
|
||||
s.newInput = nil
|
||||
}
|
||||
}
|
||||
|
||||
// Invoke assembly and send results back
|
||||
func (s *md5Server) blocks(lanes []blockInput) {
|
||||
if len(lanes) < useScalarBelow {
|
||||
// Use scalar routine when below this many lanes
|
||||
switch len(lanes) {
|
||||
case 0:
|
||||
case 1:
|
||||
lane := lanes[0]
|
||||
var d digest
|
||||
a, ok := s.digests[lane.uid]
|
||||
if ok {
|
||||
d.s[0] = binary.LittleEndian.Uint32(a[0:4])
|
||||
d.s[1] = binary.LittleEndian.Uint32(a[4:8])
|
||||
d.s[2] = binary.LittleEndian.Uint32(a[8:12])
|
||||
d.s[3] = binary.LittleEndian.Uint32(a[12:16])
|
||||
} else {
|
||||
d.s[0] = init0
|
||||
d.s[1] = init1
|
||||
d.s[2] = init2
|
||||
d.s[3] = init3
|
||||
}
|
||||
if len(lane.msg) > 0 {
|
||||
// Update...
|
||||
blockScalar(&d.s, lane.msg)
|
||||
}
|
||||
dig := [Size]byte{}
|
||||
binary.LittleEndian.PutUint32(dig[0:], d.s[0])
|
||||
binary.LittleEndian.PutUint32(dig[4:], d.s[1])
|
||||
binary.LittleEndian.PutUint32(dig[8:], d.s[2])
|
||||
binary.LittleEndian.PutUint32(dig[12:], d.s[3])
|
||||
s.digests[lane.uid] = dig
|
||||
|
||||
if lane.msg != nil {
|
||||
s.buffers <- lane.msg
|
||||
}
|
||||
lanes[0] = blockInput{}
|
||||
|
||||
default:
|
||||
s.wg.Add(len(lanes))
|
||||
var results [useScalarBelow]digest
|
||||
for i := range lanes {
|
||||
lane := lanes[i]
|
||||
go func(i int) {
|
||||
var d digest
|
||||
defer s.wg.Done()
|
||||
a, ok := s.digests[lane.uid]
|
||||
if ok {
|
||||
d.s[0] = binary.LittleEndian.Uint32(a[0:4])
|
||||
d.s[1] = binary.LittleEndian.Uint32(a[4:8])
|
||||
d.s[2] = binary.LittleEndian.Uint32(a[8:12])
|
||||
d.s[3] = binary.LittleEndian.Uint32(a[12:16])
|
||||
} else {
|
||||
d.s[0] = init0
|
||||
d.s[1] = init1
|
||||
d.s[2] = init2
|
||||
d.s[3] = init3
|
||||
}
|
||||
if len(lane.msg) == 0 {
|
||||
results[i] = d
|
||||
return
|
||||
}
|
||||
// Update...
|
||||
blockScalar(&d.s, lane.msg)
|
||||
results[i] = d
|
||||
}(i)
|
||||
}
|
||||
s.wg.Wait()
|
||||
for i, lane := range lanes {
|
||||
dig := [Size]byte{}
|
||||
binary.LittleEndian.PutUint32(dig[0:], results[i].s[0])
|
||||
binary.LittleEndian.PutUint32(dig[4:], results[i].s[1])
|
||||
binary.LittleEndian.PutUint32(dig[8:], results[i].s[2])
|
||||
binary.LittleEndian.PutUint32(dig[12:], results[i].s[3])
|
||||
s.digests[lane.uid] = dig
|
||||
|
||||
if lane.msg != nil {
|
||||
s.buffers <- lane.msg
|
||||
}
|
||||
lanes[i] = blockInput{}
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
inputs := [16][]byte{}
|
||||
for i := range lanes {
|
||||
inputs[i] = lanes[i].msg
|
||||
}
|
||||
|
||||
// Collect active digests...
|
||||
state := s.getDigests(lanes)
|
||||
// Process all lanes...
|
||||
s.blockMd5_x16(&state, inputs, len(lanes) <= 8)
|
||||
|
||||
for i, lane := range lanes {
|
||||
uid := lane.uid
|
||||
dig := [Size]byte{}
|
||||
binary.LittleEndian.PutUint32(dig[0:], state.v0[i])
|
||||
binary.LittleEndian.PutUint32(dig[4:], state.v1[i])
|
||||
binary.LittleEndian.PutUint32(dig[8:], state.v2[i])
|
||||
binary.LittleEndian.PutUint32(dig[12:], state.v3[i])
|
||||
|
||||
s.digests[uid] = dig
|
||||
if lane.msg != nil {
|
||||
s.buffers <- lane.msg
|
||||
}
|
||||
lanes[i] = blockInput{}
|
||||
}
|
||||
}
|
||||
|
||||
func (s *md5Server) getDigests(lanes []blockInput) (d digest16) {
|
||||
for i, lane := range lanes {
|
||||
a, ok := s.digests[lane.uid]
|
||||
if ok {
|
||||
d.v0[i] = binary.LittleEndian.Uint32(a[0:4])
|
||||
d.v1[i] = binary.LittleEndian.Uint32(a[4:8])
|
||||
d.v2[i] = binary.LittleEndian.Uint32(a[8:12])
|
||||
d.v3[i] = binary.LittleEndian.Uint32(a[12:16])
|
||||
} else {
|
||||
d.v0[i] = init0
|
||||
d.v1[i] = init1
|
||||
d.v2[i] = init2
|
||||
d.v3[i] = init3
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
//+build !amd64 appengine !gc noasm
|
||||
|
||||
// Copyright (c) 2020 MinIO Inc. All rights reserved.
|
||||
// Use of this source code is governed by a license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
package md5simd
|
||||
|
||||
// NewServer - Create new object for parallel processing handling
|
||||
func NewServer() *fallbackServer {
|
||||
return &fallbackServer{}
|
||||
}
|
||||
+85
@@ -0,0 +1,85 @@
|
||||
//+build !noasm,!appengine,gc
|
||||
|
||||
// Copyright (c) 2020 MinIO Inc. All rights reserved.
|
||||
// Use of this source code is governed by a license that can be
|
||||
// found in the LICENSE file.
|
||||
|
||||
package md5simd
|
||||
|
||||
// Helper struct for sorting blocks based on length
|
||||
type lane struct {
|
||||
len uint
|
||||
pos uint
|
||||
}
|
||||
|
||||
type digest struct {
|
||||
s [4]uint32
|
||||
}
|
||||
|
||||
// Helper struct for generating number of rounds in combination with mask for valid lanes
|
||||
type maskRounds struct {
|
||||
mask uint64
|
||||
rounds uint64
|
||||
}
|
||||
|
||||
func generateMaskAndRounds8(input [8][]byte, mr *[8]maskRounds) (rounds int) {
|
||||
// Sort on blocks length small to large
|
||||
var sorted [8]lane
|
||||
for c, inpt := range input[:] {
|
||||
sorted[c] = lane{uint(len(inpt)), uint(c)}
|
||||
for i := c - 1; i >= 0; i-- {
|
||||
// swap so largest is at the end...
|
||||
if sorted[i].len > sorted[i+1].len {
|
||||
sorted[i], sorted[i+1] = sorted[i+1], sorted[i]
|
||||
continue
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
// Create mask array including 'rounds' (of processing blocks of 64 bytes) between masks
|
||||
m, round := uint64(0xff), uint64(0)
|
||||
|
||||
for _, s := range sorted[:] {
|
||||
if s.len > 0 {
|
||||
if uint64(s.len)>>6 > round {
|
||||
mr[rounds] = maskRounds{m, (uint64(s.len) >> 6) - round}
|
||||
rounds++
|
||||
}
|
||||
round = uint64(s.len) >> 6
|
||||
}
|
||||
m = m & ^(1 << uint(s.pos))
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
func generateMaskAndRounds16(input [16][]byte, mr *[16]maskRounds) (rounds int) {
|
||||
// Sort on blocks length small to large
|
||||
var sorted [16]lane
|
||||
for c, inpt := range input[:] {
|
||||
sorted[c] = lane{uint(len(inpt)), uint(c)}
|
||||
for i := c - 1; i >= 0; i-- {
|
||||
// swap so largest is at the end...
|
||||
if sorted[i].len > sorted[i+1].len {
|
||||
sorted[i], sorted[i+1] = sorted[i+1], sorted[i]
|
||||
continue
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
// Create mask array including 'rounds' (of processing blocks of 64 bytes) between masks
|
||||
m, round := uint64(0xffff), uint64(0)
|
||||
|
||||
for _, s := range sorted[:] {
|
||||
if s.len > 0 {
|
||||
if uint64(s.len)>>6 > round {
|
||||
mr[rounds] = maskRounds{m, (uint64(s.len) >> 6) - round}
|
||||
rounds++
|
||||
}
|
||||
round = uint64(s.len) >> 6
|
||||
}
|
||||
m = m & ^(1 << uint(s.pos))
|
||||
}
|
||||
return
|
||||
}
|
||||
+63
@@ -0,0 +1,63 @@
|
||||
package md5simd
|
||||
|
||||
import (
|
||||
"crypto/md5"
|
||||
"hash"
|
||||
"sync"
|
||||
)
|
||||
|
||||
const (
|
||||
// The blocksize of MD5 in bytes.
|
||||
BlockSize = 64
|
||||
|
||||
// The size of an MD5 checksum in bytes.
|
||||
Size = 16
|
||||
|
||||
// internalBlockSize is the internal block size.
|
||||
internalBlockSize = 32 << 10
|
||||
)
|
||||
|
||||
type Server interface {
|
||||
NewHash() Hasher
|
||||
Close()
|
||||
}
|
||||
|
||||
type Hasher interface {
|
||||
hash.Hash
|
||||
Close()
|
||||
}
|
||||
|
||||
// StdlibHasher returns a Hasher that uses the stdlib for hashing.
|
||||
// Used hashers are stored in a pool for fast reuse.
|
||||
func StdlibHasher() Hasher {
|
||||
return &md5Wrapper{Hash: md5Pool.New().(hash.Hash)}
|
||||
}
|
||||
|
||||
// md5Wrapper is a wrapper around the builtin hasher.
|
||||
type md5Wrapper struct {
|
||||
hash.Hash
|
||||
}
|
||||
|
||||
var md5Pool = sync.Pool{New: func() interface{} {
|
||||
return md5.New()
|
||||
}}
|
||||
|
||||
// fallbackServer - Fallback when no assembly is available.
|
||||
type fallbackServer struct {
|
||||
}
|
||||
|
||||
// NewHash -- return regular Golang md5 hashing from crypto
|
||||
func (s *fallbackServer) NewHash() Hasher {
|
||||
return &md5Wrapper{Hash: md5Pool.New().(hash.Hash)}
|
||||
}
|
||||
|
||||
func (s *fallbackServer) Close() {
|
||||
}
|
||||
|
||||
func (m *md5Wrapper) Close() {
|
||||
if m.Hash != nil {
|
||||
m.Reset()
|
||||
md5Pool.Put(m.Hash)
|
||||
m.Hash = nil
|
||||
}
|
||||
}
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
// Code generated by command: go run gen.go -out ../md5block_amd64.s -stubs ../md5block_amd64.go -pkg=md5simd. DO NOT EDIT.
|
||||
|
||||
// +build !appengine
|
||||
// +build !noasm
|
||||
// +build gc
|
||||
|
||||
package md5simd
|
||||
|
||||
// Encode p to digest
|
||||
//go:noescape
|
||||
func blockScalar(dig *[4]uint32, p []byte)
|
||||
+714
@@ -0,0 +1,714 @@
|
||||
// Code generated by command: go run gen.go -out ../md5block_amd64.s -stubs ../md5block_amd64.go -pkg=md5simd. DO NOT EDIT.
|
||||
|
||||
// +build !appengine
|
||||
// +build !noasm
|
||||
// +build gc
|
||||
|
||||
// func blockScalar(dig *[4]uint32, p []byte)
|
||||
TEXT ·blockScalar(SB), $0-32
|
||||
MOVQ p_len+16(FP), AX
|
||||
MOVQ dig+0(FP), CX
|
||||
MOVQ p_base+8(FP), DX
|
||||
SHRQ $0x06, AX
|
||||
SHLQ $0x06, AX
|
||||
LEAQ (DX)(AX*1), AX
|
||||
CMPQ DX, AX
|
||||
JEQ end
|
||||
MOVL (CX), BX
|
||||
MOVL 4(CX), BP
|
||||
MOVL 8(CX), SI
|
||||
MOVL 12(CX), CX
|
||||
MOVL $0xffffffff, DI
|
||||
|
||||
loop:
|
||||
MOVL (DX), R8
|
||||
MOVL CX, R9
|
||||
MOVL BX, R10
|
||||
MOVL BP, R11
|
||||
MOVL SI, R12
|
||||
MOVL CX, R13
|
||||
|
||||
// ROUND1
|
||||
XORL SI, R9
|
||||
ADDL $0xd76aa478, BX
|
||||
ADDL R8, BX
|
||||
ANDL BP, R9
|
||||
XORL CX, R9
|
||||
MOVL 4(DX), R8
|
||||
ADDL R9, BX
|
||||
ROLL $0x07, BX
|
||||
MOVL SI, R9
|
||||
ADDL BP, BX
|
||||
XORL BP, R9
|
||||
ADDL $0xe8c7b756, CX
|
||||
ADDL R8, CX
|
||||
ANDL BX, R9
|
||||
XORL SI, R9
|
||||
MOVL 8(DX), R8
|
||||
ADDL R9, CX
|
||||
ROLL $0x0c, CX
|
||||
MOVL BP, R9
|
||||
ADDL BX, CX
|
||||
XORL BX, R9
|
||||
ADDL $0x242070db, SI
|
||||
ADDL R8, SI
|
||||
ANDL CX, R9
|
||||
XORL BP, R9
|
||||
MOVL 12(DX), R8
|
||||
ADDL R9, SI
|
||||
ROLL $0x11, SI
|
||||
MOVL BX, R9
|
||||
ADDL CX, SI
|
||||
XORL CX, R9
|
||||
ADDL $0xc1bdceee, BP
|
||||
ADDL R8, BP
|
||||
ANDL SI, R9
|
||||
XORL BX, R9
|
||||
MOVL 16(DX), R8
|
||||
ADDL R9, BP
|
||||
ROLL $0x16, BP
|
||||
MOVL CX, R9
|
||||
ADDL SI, BP
|
||||
XORL SI, R9
|
||||
ADDL $0xf57c0faf, BX
|
||||
ADDL R8, BX
|
||||
ANDL BP, R9
|
||||
XORL CX, R9
|
||||
MOVL 20(DX), R8
|
||||
ADDL R9, BX
|
||||
ROLL $0x07, BX
|
||||
MOVL SI, R9
|
||||
ADDL BP, BX
|
||||
XORL BP, R9
|
||||
ADDL $0x4787c62a, CX
|
||||
ADDL R8, CX
|
||||
ANDL BX, R9
|
||||
XORL SI, R9
|
||||
MOVL 24(DX), R8
|
||||
ADDL R9, CX
|
||||
ROLL $0x0c, CX
|
||||
MOVL BP, R9
|
||||
ADDL BX, CX
|
||||
XORL BX, R9
|
||||
ADDL $0xa8304613, SI
|
||||
ADDL R8, SI
|
||||
ANDL CX, R9
|
||||
XORL BP, R9
|
||||
MOVL 28(DX), R8
|
||||
ADDL R9, SI
|
||||
ROLL $0x11, SI
|
||||
MOVL BX, R9
|
||||
ADDL CX, SI
|
||||
XORL CX, R9
|
||||
ADDL $0xfd469501, BP
|
||||
ADDL R8, BP
|
||||
ANDL SI, R9
|
||||
XORL BX, R9
|
||||
MOVL 32(DX), R8
|
||||
ADDL R9, BP
|
||||
ROLL $0x16, BP
|
||||
MOVL CX, R9
|
||||
ADDL SI, BP
|
||||
XORL SI, R9
|
||||
ADDL $0x698098d8, BX
|
||||
ADDL R8, BX
|
||||
ANDL BP, R9
|
||||
XORL CX, R9
|
||||
MOVL 36(DX), R8
|
||||
ADDL R9, BX
|
||||
ROLL $0x07, BX
|
||||
MOVL SI, R9
|
||||
ADDL BP, BX
|
||||
XORL BP, R9
|
||||
ADDL $0x8b44f7af, CX
|
||||
ADDL R8, CX
|
||||
ANDL BX, R9
|
||||
XORL SI, R9
|
||||
MOVL 40(DX), R8
|
||||
ADDL R9, CX
|
||||
ROLL $0x0c, CX
|
||||
MOVL BP, R9
|
||||
ADDL BX, CX
|
||||
XORL BX, R9
|
||||
ADDL $0xffff5bb1, SI
|
||||
ADDL R8, SI
|
||||
ANDL CX, R9
|
||||
XORL BP, R9
|
||||
MOVL 44(DX), R8
|
||||
ADDL R9, SI
|
||||
ROLL $0x11, SI
|
||||
MOVL BX, R9
|
||||
ADDL CX, SI
|
||||
XORL CX, R9
|
||||
ADDL $0x895cd7be, BP
|
||||
ADDL R8, BP
|
||||
ANDL SI, R9
|
||||
XORL BX, R9
|
||||
MOVL 48(DX), R8
|
||||
ADDL R9, BP
|
||||
ROLL $0x16, BP
|
||||
MOVL CX, R9
|
||||
ADDL SI, BP
|
||||
XORL SI, R9
|
||||
ADDL $0x6b901122, BX
|
||||
ADDL R8, BX
|
||||
ANDL BP, R9
|
||||
XORL CX, R9
|
||||
MOVL 52(DX), R8
|
||||
ADDL R9, BX
|
||||
ROLL $0x07, BX
|
||||
MOVL SI, R9
|
||||
ADDL BP, BX
|
||||
XORL BP, R9
|
||||
ADDL $0xfd987193, CX
|
||||
ADDL R8, CX
|
||||
ANDL BX, R9
|
||||
XORL SI, R9
|
||||
MOVL 56(DX), R8
|
||||
ADDL R9, CX
|
||||
ROLL $0x0c, CX
|
||||
MOVL BP, R9
|
||||
ADDL BX, CX
|
||||
XORL BX, R9
|
||||
ADDL $0xa679438e, SI
|
||||
ADDL R8, SI
|
||||
ANDL CX, R9
|
||||
XORL BP, R9
|
||||
MOVL 60(DX), R8
|
||||
ADDL R9, SI
|
||||
ROLL $0x11, SI
|
||||
MOVL BX, R9
|
||||
ADDL CX, SI
|
||||
XORL CX, R9
|
||||
ADDL $0x49b40821, BP
|
||||
ADDL R8, BP
|
||||
ANDL SI, R9
|
||||
XORL BX, R9
|
||||
MOVL 4(DX), R8
|
||||
ADDL R9, BP
|
||||
ROLL $0x16, BP
|
||||
MOVL CX, R9
|
||||
ADDL SI, BP
|
||||
|
||||
// ROUND2
|
||||
MOVL CX, R9
|
||||
MOVL CX, R14
|
||||
XORL DI, R9
|
||||
ADDL $0xf61e2562, BX
|
||||
ADDL R8, BX
|
||||
ANDL BP, R14
|
||||
ANDL SI, R9
|
||||
MOVL 24(DX), R8
|
||||
ORL R9, R14
|
||||
MOVL SI, R9
|
||||
ADDL R14, BX
|
||||
MOVL SI, R14
|
||||
ROLL $0x05, BX
|
||||
ADDL BP, BX
|
||||
XORL DI, R9
|
||||
ADDL $0xc040b340, CX
|
||||
ADDL R8, CX
|
||||
ANDL BX, R14
|
||||
ANDL BP, R9
|
||||
MOVL 44(DX), R8
|
||||
ORL R9, R14
|
||||
MOVL BP, R9
|
||||
ADDL R14, CX
|
||||
MOVL BP, R14
|
||||
ROLL $0x09, CX
|
||||
ADDL BX, CX
|
||||
XORL DI, R9
|
||||
ADDL $0x265e5a51, SI
|
||||
ADDL R8, SI
|
||||
ANDL CX, R14
|
||||
ANDL BX, R9
|
||||
MOVL (DX), R8
|
||||
ORL R9, R14
|
||||
MOVL BX, R9
|
||||
ADDL R14, SI
|
||||
MOVL BX, R14
|
||||
ROLL $0x0e, SI
|
||||
ADDL CX, SI
|
||||
XORL DI, R9
|
||||
ADDL $0xe9b6c7aa, BP
|
||||
ADDL R8, BP
|
||||
ANDL SI, R14
|
||||
ANDL CX, R9
|
||||
MOVL 20(DX), R8
|
||||
ORL R9, R14
|
||||
MOVL CX, R9
|
||||
ADDL R14, BP
|
||||
MOVL CX, R14
|
||||
ROLL $0x14, BP
|
||||
ADDL SI, BP
|
||||
XORL DI, R9
|
||||
ADDL $0xd62f105d, BX
|
||||
ADDL R8, BX
|
||||
ANDL BP, R14
|
||||
ANDL SI, R9
|
||||
MOVL 40(DX), R8
|
||||
ORL R9, R14
|
||||
MOVL SI, R9
|
||||
ADDL R14, BX
|
||||
MOVL SI, R14
|
||||
ROLL $0x05, BX
|
||||
ADDL BP, BX
|
||||
XORL DI, R9
|
||||
ADDL $0x02441453, CX
|
||||
ADDL R8, CX
|
||||
ANDL BX, R14
|
||||
ANDL BP, R9
|
||||
MOVL 60(DX), R8
|
||||
ORL R9, R14
|
||||
MOVL BP, R9
|
||||
ADDL R14, CX
|
||||
MOVL BP, R14
|
||||
ROLL $0x09, CX
|
||||
ADDL BX, CX
|
||||
XORL DI, R9
|
||||
ADDL $0xd8a1e681, SI
|
||||
ADDL R8, SI
|
||||
ANDL CX, R14
|
||||
ANDL BX, R9
|
||||
MOVL 16(DX), R8
|
||||
ORL R9, R14
|
||||
MOVL BX, R9
|
||||
ADDL R14, SI
|
||||
MOVL BX, R14
|
||||
ROLL $0x0e, SI
|
||||
ADDL CX, SI
|
||||
XORL DI, R9
|
||||
ADDL $0xe7d3fbc8, BP
|
||||
ADDL R8, BP
|
||||
ANDL SI, R14
|
||||
ANDL CX, R9
|
||||
MOVL 36(DX), R8
|
||||
ORL R9, R14
|
||||
MOVL CX, R9
|
||||
ADDL R14, BP
|
||||
MOVL CX, R14
|
||||
ROLL $0x14, BP
|
||||
ADDL SI, BP
|
||||
XORL DI, R9
|
||||
ADDL $0x21e1cde6, BX
|
||||
ADDL R8, BX
|
||||
ANDL BP, R14
|
||||
ANDL SI, R9
|
||||
MOVL 56(DX), R8
|
||||
ORL R9, R14
|
||||
MOVL SI, R9
|
||||
ADDL R14, BX
|
||||
MOVL SI, R14
|
||||
ROLL $0x05, BX
|
||||
ADDL BP, BX
|
||||
XORL DI, R9
|
||||
ADDL $0xc33707d6, CX
|
||||
ADDL R8, CX
|
||||
ANDL BX, R14
|
||||
ANDL BP, R9
|
||||
MOVL 12(DX), R8
|
||||
ORL R9, R14
|
||||
MOVL BP, R9
|
||||
ADDL R14, CX
|
||||
MOVL BP, R14
|
||||
ROLL $0x09, CX
|
||||
ADDL BX, CX
|
||||
XORL DI, R9
|
||||
ADDL $0xf4d50d87, SI
|
||||
ADDL R8, SI
|
||||
ANDL CX, R14
|
||||
ANDL BX, R9
|
||||
MOVL 32(DX), R8
|
||||
ORL R9, R14
|
||||
MOVL BX, R9
|
||||
ADDL R14, SI
|
||||
MOVL BX, R14
|
||||
ROLL $0x0e, SI
|
||||
ADDL CX, SI
|
||||
XORL DI, R9
|
||||
ADDL $0x455a14ed, BP
|
||||
ADDL R8, BP
|
||||
ANDL SI, R14
|
||||
ANDL CX, R9
|
||||
MOVL 52(DX), R8
|
||||
ORL R9, R14
|
||||
MOVL CX, R9
|
||||
ADDL R14, BP
|
||||
MOVL CX, R14
|
||||
ROLL $0x14, BP
|
||||
ADDL SI, BP
|
||||
XORL DI, R9
|
||||
ADDL $0xa9e3e905, BX
|
||||
ADDL R8, BX
|
||||
ANDL BP, R14
|
||||
ANDL SI, R9
|
||||
MOVL 8(DX), R8
|
||||
ORL R9, R14
|
||||
MOVL SI, R9
|
||||
ADDL R14, BX
|
||||
MOVL SI, R14
|
||||
ROLL $0x05, BX
|
||||
ADDL BP, BX
|
||||
XORL DI, R9
|
||||
ADDL $0xfcefa3f8, CX
|
||||
ADDL R8, CX
|
||||
ANDL BX, R14
|
||||
ANDL BP, R9
|
||||
MOVL 28(DX), R8
|
||||
ORL R9, R14
|
||||
MOVL BP, R9
|
||||
ADDL R14, CX
|
||||
MOVL BP, R14
|
||||
ROLL $0x09, CX
|
||||
ADDL BX, CX
|
||||
XORL DI, R9
|
||||
ADDL $0x676f02d9, SI
|
||||
ADDL R8, SI
|
||||
ANDL CX, R14
|
||||
ANDL BX, R9
|
||||
MOVL 48(DX), R8
|
||||
ORL R9, R14
|
||||
MOVL BX, R9
|
||||
ADDL R14, SI
|
||||
MOVL BX, R14
|
||||
ROLL $0x0e, SI
|
||||
ADDL CX, SI
|
||||
XORL DI, R9
|
||||
ADDL $0x8d2a4c8a, BP
|
||||
ADDL R8, BP
|
||||
ANDL SI, R14
|
||||
ANDL CX, R9
|
||||
MOVL 20(DX), R8
|
||||
ORL R9, R14
|
||||
MOVL CX, R9
|
||||
ADDL R14, BP
|
||||
MOVL CX, R14
|
||||
ROLL $0x14, BP
|
||||
ADDL SI, BP
|
||||
|
||||
// ROUND3
|
||||
MOVL SI, R9
|
||||
ADDL $0xfffa3942, BX
|
||||
ADDL R8, BX
|
||||
MOVL 32(DX), R8
|
||||
XORL CX, R9
|
||||
XORL BP, R9
|
||||
ADDL R9, BX
|
||||
ROLL $0x04, BX
|
||||
MOVL BP, R9
|
||||
ADDL BP, BX
|
||||
ADDL $0x8771f681, CX
|
||||
ADDL R8, CX
|
||||
MOVL 44(DX), R8
|
||||
XORL SI, R9
|
||||
XORL BX, R9
|
||||
ADDL R9, CX
|
||||
ROLL $0x0b, CX
|
||||
MOVL BX, R9
|
||||
ADDL BX, CX
|
||||
ADDL $0x6d9d6122, SI
|
||||
ADDL R8, SI
|
||||
MOVL 56(DX), R8
|
||||
XORL BP, R9
|
||||
XORL CX, R9
|
||||
ADDL R9, SI
|
||||
ROLL $0x10, SI
|
||||
MOVL CX, R9
|
||||
ADDL CX, SI
|
||||
ADDL $0xfde5380c, BP
|
||||
ADDL R8, BP
|
||||
MOVL 4(DX), R8
|
||||
XORL BX, R9
|
||||
XORL SI, R9
|
||||
ADDL R9, BP
|
||||
ROLL $0x17, BP
|
||||
MOVL SI, R9
|
||||
ADDL SI, BP
|
||||
ADDL $0xa4beea44, BX
|
||||
ADDL R8, BX
|
||||
MOVL 16(DX), R8
|
||||
XORL CX, R9
|
||||
XORL BP, R9
|
||||
ADDL R9, BX
|
||||
ROLL $0x04, BX
|
||||
MOVL BP, R9
|
||||
ADDL BP, BX
|
||||
ADDL $0x4bdecfa9, CX
|
||||
ADDL R8, CX
|
||||
MOVL 28(DX), R8
|
||||
XORL SI, R9
|
||||
XORL BX, R9
|
||||
ADDL R9, CX
|
||||
ROLL $0x0b, CX
|
||||
MOVL BX, R9
|
||||
ADDL BX, CX
|
||||
ADDL $0xf6bb4b60, SI
|
||||
ADDL R8, SI
|
||||
MOVL 40(DX), R8
|
||||
XORL BP, R9
|
||||
XORL CX, R9
|
||||
ADDL R9, SI
|
||||
ROLL $0x10, SI
|
||||
MOVL CX, R9
|
||||
ADDL CX, SI
|
||||
ADDL $0xbebfbc70, BP
|
||||
ADDL R8, BP
|
||||
MOVL 52(DX), R8
|
||||
XORL BX, R9
|
||||
XORL SI, R9
|
||||
ADDL R9, BP
|
||||
ROLL $0x17, BP
|
||||
MOVL SI, R9
|
||||
ADDL SI, BP
|
||||
ADDL $0x289b7ec6, BX
|
||||
ADDL R8, BX
|
||||
MOVL (DX), R8
|
||||
XORL CX, R9
|
||||
XORL BP, R9
|
||||
ADDL R9, BX
|
||||
ROLL $0x04, BX
|
||||
MOVL BP, R9
|
||||
ADDL BP, BX
|
||||
ADDL $0xeaa127fa, CX
|
||||
ADDL R8, CX
|
||||
MOVL 12(DX), R8
|
||||
XORL SI, R9
|
||||
XORL BX, R9
|
||||
ADDL R9, CX
|
||||
ROLL $0x0b, CX
|
||||
MOVL BX, R9
|
||||
ADDL BX, CX
|
||||
ADDL $0xd4ef3085, SI
|
||||
ADDL R8, SI
|
||||
MOVL 24(DX), R8
|
||||
XORL BP, R9
|
||||
XORL CX, R9
|
||||
ADDL R9, SI
|
||||
ROLL $0x10, SI
|
||||
MOVL CX, R9
|
||||
ADDL CX, SI
|
||||
ADDL $0x04881d05, BP
|
||||
ADDL R8, BP
|
||||
MOVL 36(DX), R8
|
||||
XORL BX, R9
|
||||
XORL SI, R9
|
||||
ADDL R9, BP
|
||||
ROLL $0x17, BP
|
||||
MOVL SI, R9
|
||||
ADDL SI, BP
|
||||
ADDL $0xd9d4d039, BX
|
||||
ADDL R8, BX
|
||||
MOVL 48(DX), R8
|
||||
XORL CX, R9
|
||||
XORL BP, R9
|
||||
ADDL R9, BX
|
||||
ROLL $0x04, BX
|
||||
MOVL BP, R9
|
||||
ADDL BP, BX
|
||||
ADDL $0xe6db99e5, CX
|
||||
ADDL R8, CX
|
||||
MOVL 60(DX), R8
|
||||
XORL SI, R9
|
||||
XORL BX, R9
|
||||
ADDL R9, CX
|
||||
ROLL $0x0b, CX
|
||||
MOVL BX, R9
|
||||
ADDL BX, CX
|
||||
ADDL $0x1fa27cf8, SI
|
||||
ADDL R8, SI
|
||||
MOVL 8(DX), R8
|
||||
XORL BP, R9
|
||||
XORL CX, R9
|
||||
ADDL R9, SI
|
||||
ROLL $0x10, SI
|
||||
MOVL CX, R9
|
||||
ADDL CX, SI
|
||||
ADDL $0xc4ac5665, BP
|
||||
ADDL R8, BP
|
||||
MOVL (DX), R8
|
||||
XORL BX, R9
|
||||
XORL SI, R9
|
||||
ADDL R9, BP
|
||||
ROLL $0x17, BP
|
||||
MOVL SI, R9
|
||||
ADDL SI, BP
|
||||
|
||||
// ROUND4
|
||||
MOVL DI, R9
|
||||
XORL CX, R9
|
||||
ADDL $0xf4292244, BX
|
||||
ADDL R8, BX
|
||||
ORL BP, R9
|
||||
XORL SI, R9
|
||||
ADDL R9, BX
|
||||
MOVL 28(DX), R8
|
||||
MOVL DI, R9
|
||||
ROLL $0x06, BX
|
||||
XORL SI, R9
|
||||
ADDL BP, BX
|
||||
ADDL $0x432aff97, CX
|
||||
ADDL R8, CX
|
||||
ORL BX, R9
|
||||
XORL BP, R9
|
||||
ADDL R9, CX
|
||||
MOVL 56(DX), R8
|
||||
MOVL DI, R9
|
||||
ROLL $0x0a, CX
|
||||
XORL BP, R9
|
||||
ADDL BX, CX
|
||||
ADDL $0xab9423a7, SI
|
||||
ADDL R8, SI
|
||||
ORL CX, R9
|
||||
XORL BX, R9
|
||||
ADDL R9, SI
|
||||
MOVL 20(DX), R8
|
||||
MOVL DI, R9
|
||||
ROLL $0x0f, SI
|
||||
XORL BX, R9
|
||||
ADDL CX, SI
|
||||
ADDL $0xfc93a039, BP
|
||||
ADDL R8, BP
|
||||
ORL SI, R9
|
||||
XORL CX, R9
|
||||
ADDL R9, BP
|
||||
MOVL 48(DX), R8
|
||||
MOVL DI, R9
|
||||
ROLL $0x15, BP
|
||||
XORL CX, R9
|
||||
ADDL SI, BP
|
||||
ADDL $0x655b59c3, BX
|
||||
ADDL R8, BX
|
||||
ORL BP, R9
|
||||
XORL SI, R9
|
||||
ADDL R9, BX
|
||||
MOVL 12(DX), R8
|
||||
MOVL DI, R9
|
||||
ROLL $0x06, BX
|
||||
XORL SI, R9
|
||||
ADDL BP, BX
|
||||
ADDL $0x8f0ccc92, CX
|
||||
ADDL R8, CX
|
||||
ORL BX, R9
|
||||
XORL BP, R9
|
||||
ADDL R9, CX
|
||||
MOVL 40(DX), R8
|
||||
MOVL DI, R9
|
||||
ROLL $0x0a, CX
|
||||
XORL BP, R9
|
||||
ADDL BX, CX
|
||||
ADDL $0xffeff47d, SI
|
||||
ADDL R8, SI
|
||||
ORL CX, R9
|
||||
XORL BX, R9
|
||||
ADDL R9, SI
|
||||
MOVL 4(DX), R8
|
||||
MOVL DI, R9
|
||||
ROLL $0x0f, SI
|
||||
XORL BX, R9
|
||||
ADDL CX, SI
|
||||
ADDL $0x85845dd1, BP
|
||||
ADDL R8, BP
|
||||
ORL SI, R9
|
||||
XORL CX, R9
|
||||
ADDL R9, BP
|
||||
MOVL 32(DX), R8
|
||||
MOVL DI, R9
|
||||
ROLL $0x15, BP
|
||||
XORL CX, R9
|
||||
ADDL SI, BP
|
||||
ADDL $0x6fa87e4f, BX
|
||||
ADDL R8, BX
|
||||
ORL BP, R9
|
||||
XORL SI, R9
|
||||
ADDL R9, BX
|
||||
MOVL 60(DX), R8
|
||||
MOVL DI, R9
|
||||
ROLL $0x06, BX
|
||||
XORL SI, R9
|
||||
ADDL BP, BX
|
||||
ADDL $0xfe2ce6e0, CX
|
||||
ADDL R8, CX
|
||||
ORL BX, R9
|
||||
XORL BP, R9
|
||||
ADDL R9, CX
|
||||
MOVL 24(DX), R8
|
||||
MOVL DI, R9
|
||||
ROLL $0x0a, CX
|
||||
XORL BP, R9
|
||||
ADDL BX, CX
|
||||
ADDL $0xa3014314, SI
|
||||
ADDL R8, SI
|
||||
ORL CX, R9
|
||||
XORL BX, R9
|
||||
ADDL R9, SI
|
||||
MOVL 52(DX), R8
|
||||
MOVL DI, R9
|
||||
ROLL $0x0f, SI
|
||||
XORL BX, R9
|
||||
ADDL CX, SI
|
||||
ADDL $0x4e0811a1, BP
|
||||
ADDL R8, BP
|
||||
ORL SI, R9
|
||||
XORL CX, R9
|
||||
ADDL R9, BP
|
||||
MOVL 16(DX), R8
|
||||
MOVL DI, R9
|
||||
ROLL $0x15, BP
|
||||
XORL CX, R9
|
||||
ADDL SI, BP
|
||||
ADDL $0xf7537e82, BX
|
||||
ADDL R8, BX
|
||||
ORL BP, R9
|
||||
XORL SI, R9
|
||||
ADDL R9, BX
|
||||
MOVL 44(DX), R8
|
||||
MOVL DI, R9
|
||||
ROLL $0x06, BX
|
||||
XORL SI, R9
|
||||
ADDL BP, BX
|
||||
ADDL $0xbd3af235, CX
|
||||
ADDL R8, CX
|
||||
ORL BX, R9
|
||||
XORL BP, R9
|
||||
ADDL R9, CX
|
||||
MOVL 8(DX), R8
|
||||
MOVL DI, R9
|
||||
ROLL $0x0a, CX
|
||||
XORL BP, R9
|
||||
ADDL BX, CX
|
||||
ADDL $0x2ad7d2bb, SI
|
||||
ADDL R8, SI
|
||||
ORL CX, R9
|
||||
XORL BX, R9
|
||||
ADDL R9, SI
|
||||
MOVL 36(DX), R8
|
||||
MOVL DI, R9
|
||||
ROLL $0x0f, SI
|
||||
XORL BX, R9
|
||||
ADDL CX, SI
|
||||
ADDL $0xeb86d391, BP
|
||||
ADDL R8, BP
|
||||
ORL SI, R9
|
||||
XORL CX, R9
|
||||
ADDL R9, BP
|
||||
ROLL $0x15, BP
|
||||
ADDL SI, BP
|
||||
ADDL R10, BX
|
||||
ADDL R11, BP
|
||||
ADDL R12, SI
|
||||
ADDL R13, CX
|
||||
|
||||
// Prepare next loop
|
||||
ADDQ $0x40, DX
|
||||
CMPQ DX, AX
|
||||
JB loop
|
||||
|
||||
// Write output
|
||||
MOVQ dig+0(FP), AX
|
||||
MOVL BX, (AX)
|
||||
MOVL BP, 4(AX)
|
||||
MOVL SI, 8(AX)
|
||||
MOVL CX, 12(AX)
|
||||
|
||||
end:
|
||||
RET
|
||||
@@ -0,0 +1,6 @@
|
||||
*~
|
||||
*.test
|
||||
validator
|
||||
golangci-lint
|
||||
functional_tests
|
||||
.idea
|
||||
+27
@@ -0,0 +1,27 @@
|
||||
linters-settings:
|
||||
misspell:
|
||||
locale: US
|
||||
|
||||
linters:
|
||||
disable-all: true
|
||||
enable:
|
||||
- typecheck
|
||||
- goimports
|
||||
- misspell
|
||||
- revive
|
||||
- govet
|
||||
- ineffassign
|
||||
- gosimple
|
||||
- unused
|
||||
- gocritic
|
||||
|
||||
issues:
|
||||
exclude-use-default: false
|
||||
exclude:
|
||||
# todo fix these when we get enough time.
|
||||
- "singleCaseSwitch: should rewrite switch statement to if statement"
|
||||
- "unlambda: replace"
|
||||
- "captLocal:"
|
||||
- "ifElseChain:"
|
||||
- "elseif:"
|
||||
- "should have a package comment"
|
||||
+1
@@ -0,0 +1 @@
|
||||
minio-go.min.io
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
### Developer Guidelines
|
||||
|
||||
``minio-go`` welcomes your contribution. To make the process as seamless as possible, we ask for the following:
|
||||
|
||||
* Go ahead and fork the project and make your changes. We encourage pull requests to discuss code changes.
|
||||
- Fork it
|
||||
- Create your feature branch (git checkout -b my-new-feature)
|
||||
- Commit your changes (git commit -am 'Add some feature')
|
||||
- Push to the branch (git push origin my-new-feature)
|
||||
- Create new Pull Request
|
||||
|
||||
* When you're ready to create a pull request, be sure to:
|
||||
- Have test cases for the new code. If you have questions about how to do it, please ask in your pull request.
|
||||
- Run `go fmt`
|
||||
- Squash your commits into a single commit. `git rebase -i`. It's okay to force update your pull request.
|
||||
- Make sure `go test -race ./...` and `go build` completes.
|
||||
NOTE: go test runs functional tests and requires you to have a AWS S3 account. Set them as environment variables
|
||||
``ACCESS_KEY`` and ``SECRET_KEY``. To run shorter version of the tests please use ``go test -short -race ./...``
|
||||
|
||||
* Read [Effective Go](https://github.com/golang/go/wiki/CodeReviewComments) article from Golang project
|
||||
- `minio-go` project is strictly conformant with Golang style
|
||||
- if you happen to observe offending code, please feel free to send a pull request
|
||||
+1101
File diff suppressed because it is too large
Load Diff
+202
@@ -0,0 +1,202 @@
|
||||
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
# For maintainers only
|
||||
|
||||
## Responsibilities
|
||||
|
||||
Please go through this link [Maintainer Responsibility](https://gist.github.com/abperiasamy/f4d9b31d3186bbd26522)
|
||||
|
||||
### Making new releases
|
||||
Tag and sign your release commit, additionally this step requires you to have access to MinIO's trusted private key.
|
||||
```sh
|
||||
$ export GNUPGHOME=/media/${USER}/minio/trusted
|
||||
$ git tag -s 4.0.0
|
||||
$ git push
|
||||
$ git push --tags
|
||||
```
|
||||
|
||||
### Update version
|
||||
Once release has been made update `libraryVersion` constant in `api.go` to next to be released version.
|
||||
|
||||
```sh
|
||||
$ grep libraryVersion api.go
|
||||
libraryVersion = "4.0.1"
|
||||
```
|
||||
|
||||
Commit your changes
|
||||
```
|
||||
$ git commit -a -m "Update version for next release" --author "MinIO Trusted <trusted@min.io>"
|
||||
```
|
||||
|
||||
### Announce
|
||||
Announce new release by adding release notes at https://github.com/minio/minio-go/releases from `trusted@min.io` account. Release notes requires two sections `highlights` and `changelog`. Highlights is a bulleted list of salient features in this release and Changelog contains list of all commits since the last release.
|
||||
|
||||
To generate `changelog`
|
||||
```sh
|
||||
$ git log --no-color --pretty=format:'-%d %s (%cr) <%an>' <last_release_tag>..<latest_release_tag>
|
||||
```
|
||||
+42
@@ -0,0 +1,42 @@
|
||||
GOPATH := $(shell go env GOPATH)
|
||||
TMPDIR := $(shell mktemp -d)
|
||||
|
||||
all: checks
|
||||
|
||||
.PHONY: examples docs
|
||||
|
||||
checks: lint vet test examples functional-test
|
||||
|
||||
lint:
|
||||
@mkdir -p ${GOPATH}/bin
|
||||
@echo "Installing golangci-lint" && curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(GOPATH)/bin
|
||||
@echo "Running $@ check"
|
||||
@GO111MODULE=on ${GOPATH}/bin/golangci-lint cache clean
|
||||
@GO111MODULE=on ${GOPATH}/bin/golangci-lint run --timeout=5m --config ./.golangci.yml
|
||||
|
||||
vet:
|
||||
@GO111MODULE=on go vet ./...
|
||||
@echo "Installing staticcheck" && go install honnef.co/go/tools/cmd/staticcheck@latest
|
||||
${GOPATH}/bin/staticcheck -tests=false -checks="all,-ST1000,-ST1003,-ST1016,-ST1020,-ST1021,-ST1022,-ST1023,-ST1005"
|
||||
|
||||
test:
|
||||
@GO111MODULE=on SERVER_ENDPOINT=localhost:9000 ACCESS_KEY=minioadmin SECRET_KEY=minioadmin ENABLE_HTTPS=1 MINT_MODE=full go test -race -v ./...
|
||||
|
||||
examples:
|
||||
@echo "Building s3 examples"
|
||||
@cd ./examples/s3 && $(foreach v,$(wildcard examples/s3/*.go),go build -mod=mod -o ${TMPDIR}/$(basename $(v)) $(notdir $(v)) || exit 1;)
|
||||
@echo "Building minio examples"
|
||||
@cd ./examples/minio && $(foreach v,$(wildcard examples/minio/*.go),go build -mod=mod -o ${TMPDIR}/$(basename $(v)) $(notdir $(v)) || exit 1;)
|
||||
|
||||
functional-test:
|
||||
@GO111MODULE=on go build -race functional_tests.go
|
||||
@SERVER_ENDPOINT=localhost:9000 ACCESS_KEY=minioadmin SECRET_KEY=minioadmin ENABLE_HTTPS=1 MINT_MODE=full ./functional_tests
|
||||
|
||||
functional-test-notls:
|
||||
@GO111MODULE=on go build -race functional_tests.go
|
||||
@SERVER_ENDPOINT=localhost:9000 ACCESS_KEY=minioadmin SECRET_KEY=minioadmin ENABLE_HTTPS=0 MINT_MODE=full ./functional_tests
|
||||
|
||||
clean:
|
||||
@echo "Cleaning up all the generated files"
|
||||
@find . -name '*.test' | xargs rm -fv
|
||||
@find . -name '*~' | xargs rm -fv
|
||||
+9
@@ -0,0 +1,9 @@
|
||||
MinIO Cloud Storage, (C) 2014-2020 MinIO, Inc.
|
||||
|
||||
This product includes software developed at MinIO, Inc.
|
||||
(https://min.io/).
|
||||
|
||||
The MinIO project contains unmodified/modified subcomponents too with
|
||||
separate copyright notices and license terms. Your use of the source
|
||||
code for these subcomponents is subject to the terms and conditions
|
||||
of Apache License Version 2.0
|
||||
+312
@@ -0,0 +1,312 @@
|
||||
# MinIO Go Client SDK for Amazon S3 Compatible Cloud Storage [](https://slack.min.io) [](https://sourcegraph.com/github.com/minio/minio-go?badge) [](https://github.com/minio/minio-go/blob/master/LICENSE)
|
||||
|
||||
The MinIO Go Client SDK provides straightforward APIs to access any Amazon S3 compatible object storage.
|
||||
|
||||
This Quickstart Guide covers how to install the MinIO client SDK, connect to MinIO, and create a sample file uploader.
|
||||
For a complete list of APIs and examples, see the [godoc documentation](https://pkg.go.dev/github.com/minio/minio-go/v7) or [Go Client API Reference](https://min.io/docs/minio/linux/developers/go/API.html).
|
||||
|
||||
These examples presume a working [Go development environment](https://golang.org/doc/install) and the [MinIO `mc` command line tool](https://min.io/docs/minio/linux/reference/minio-mc.html).
|
||||
|
||||
## Download from Github
|
||||
|
||||
From your project directory:
|
||||
|
||||
```sh
|
||||
go get github.com/minio/minio-go/v7
|
||||
```
|
||||
|
||||
## Initialize a MinIO Client Object
|
||||
|
||||
The MinIO client requires the following parameters to connect to an Amazon S3 compatible object storage:
|
||||
|
||||
| Parameter | Description |
|
||||
| ----------------- | ---------------------------------------------------------- |
|
||||
| `endpoint` | URL to object storage service. |
|
||||
| `_minio.Options_` | All the options such as credentials, custom transport etc. |
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"log"
|
||||
|
||||
"github.com/minio/minio-go/v7"
|
||||
"github.com/minio/minio-go/v7/pkg/credentials"
|
||||
)
|
||||
|
||||
func main() {
|
||||
endpoint := "play.min.io"
|
||||
accessKeyID := "Q3AM3UQ867SPQQA43P2F"
|
||||
secretAccessKey := "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG"
|
||||
useSSL := true
|
||||
|
||||
// Initialize minio client object.
|
||||
minioClient, err := minio.New(endpoint, &minio.Options{
|
||||
Creds: credentials.NewStaticV4(accessKeyID, secretAccessKey, ""),
|
||||
Secure: useSSL,
|
||||
})
|
||||
if err != nil {
|
||||
log.Fatalln(err)
|
||||
}
|
||||
|
||||
log.Printf("%#v\n", minioClient) // minioClient is now set up
|
||||
}
|
||||
```
|
||||
|
||||
## Example - File Uploader
|
||||
|
||||
This sample code connects to an object storage server, creates a bucket, and uploads a file to the bucket.
|
||||
It uses the MinIO `play` server, a public MinIO cluster located at [https://play.min.io](https://play.min.io).
|
||||
|
||||
The `play` server runs the latest stable version of MinIO and may be used for testing and development.
|
||||
The access credentials shown in this example are open to the public and all data uploaded to `play` should be considered public and non-protected.
|
||||
|
||||
### FileUploader.go
|
||||
|
||||
This example does the following:
|
||||
|
||||
- Connects to the MinIO `play` server using the provided credentials.
|
||||
- Creates a bucket named `testbucket`.
|
||||
- Uploads a file named `testdata` from `/tmp`.
|
||||
- Verifies the file was created using `mc ls`.
|
||||
|
||||
```go
|
||||
// FileUploader.go MinIO example
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log"
|
||||
|
||||
"github.com/minio/minio-go/v7"
|
||||
"github.com/minio/minio-go/v7/pkg/credentials"
|
||||
)
|
||||
|
||||
func main() {
|
||||
ctx := context.Background()
|
||||
endpoint := "play.min.io"
|
||||
accessKeyID := "Q3AM3UQ867SPQQA43P2F"
|
||||
secretAccessKey := "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG"
|
||||
useSSL := true
|
||||
|
||||
// Initialize minio client object.
|
||||
minioClient, err := minio.New(endpoint, &minio.Options{
|
||||
Creds: credentials.NewStaticV4(accessKeyID, secretAccessKey, ""),
|
||||
Secure: useSSL,
|
||||
})
|
||||
if err != nil {
|
||||
log.Fatalln(err)
|
||||
}
|
||||
|
||||
// Make a new bucket called testbucket.
|
||||
bucketName := "testbucket"
|
||||
location := "us-east-1"
|
||||
|
||||
err = minioClient.MakeBucket(ctx, bucketName, minio.MakeBucketOptions{Region: location})
|
||||
if err != nil {
|
||||
// Check to see if we already own this bucket (which happens if you run this twice)
|
||||
exists, errBucketExists := minioClient.BucketExists(ctx, bucketName)
|
||||
if errBucketExists == nil && exists {
|
||||
log.Printf("We already own %s\n", bucketName)
|
||||
} else {
|
||||
log.Fatalln(err)
|
||||
}
|
||||
} else {
|
||||
log.Printf("Successfully created %s\n", bucketName)
|
||||
}
|
||||
|
||||
// Upload the test file
|
||||
// Change the value of filePath if the file is in another location
|
||||
objectName := "testdata"
|
||||
filePath := "/tmp/testdata"
|
||||
contentType := "application/octet-stream"
|
||||
|
||||
// Upload the test file with FPutObject
|
||||
info, err := minioClient.FPutObject(ctx, bucketName, objectName, filePath, minio.PutObjectOptions{ContentType: contentType})
|
||||
if err != nil {
|
||||
log.Fatalln(err)
|
||||
}
|
||||
|
||||
log.Printf("Successfully uploaded %s of size %d\n", objectName, info.Size)
|
||||
}
|
||||
```
|
||||
|
||||
**1. Create a test file containing data:**
|
||||
|
||||
You can do this with `dd` on Linux or macOS systems:
|
||||
|
||||
```sh
|
||||
dd if=/dev/urandom of=/tmp/testdata bs=2048 count=10
|
||||
```
|
||||
|
||||
or `fsutil` on Windows:
|
||||
|
||||
```sh
|
||||
fsutil file createnew "C:\Users\<username>\Desktop\sample.txt" 20480
|
||||
```
|
||||
|
||||
**2. Run FileUploader with the following commands:**
|
||||
|
||||
```sh
|
||||
go mod init example/FileUploader
|
||||
go get github.com/minio/minio-go/v7
|
||||
go get github.com/minio/minio-go/v7/pkg/credentials
|
||||
go run FileUploader.go
|
||||
```
|
||||
|
||||
The output resembles the following:
|
||||
|
||||
```sh
|
||||
2023/11/01 14:27:55 Successfully created testbucket
|
||||
2023/11/01 14:27:55 Successfully uploaded testdata of size 20480
|
||||
```
|
||||
|
||||
**3. Verify the Uploaded File With `mc ls`:**
|
||||
|
||||
```sh
|
||||
mc ls play/testbucket
|
||||
[2023-11-01 14:27:55 UTC] 20KiB STANDARD TestDataFile
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
The full API Reference is available here.
|
||||
|
||||
* [Complete API Reference](https://min.io/docs/minio/linux/developers/go/API.html)
|
||||
|
||||
### API Reference : Bucket Operations
|
||||
|
||||
* [`MakeBucket`](https://min.io/docs/minio/linux/developers/go/API.html#MakeBucket)
|
||||
* [`ListBuckets`](https://min.io/docs/minio/linux/developers/go/API.html#ListBuckets)
|
||||
* [`BucketExists`](https://min.io/docs/minio/linux/developers/go/API.html#BucketExists)
|
||||
* [`RemoveBucket`](https://min.io/docs/minio/linux/developers/go/API.html#RemoveBucket)
|
||||
* [`ListObjects`](https://min.io/docs/minio/linux/developers/go/API.html#ListObjects)
|
||||
* [`ListIncompleteUploads`](https://min.io/docs/minio/linux/developers/go/API.html#ListIncompleteUploads)
|
||||
|
||||
### API Reference : Bucket policy Operations
|
||||
|
||||
* [`SetBucketPolicy`](https://min.io/docs/minio/linux/developers/go/API.html#SetBucketPolicy)
|
||||
* [`GetBucketPolicy`](https://min.io/docs/minio/linux/developers/go/API.html#GetBucketPolicy)
|
||||
|
||||
### API Reference : Bucket notification Operations
|
||||
|
||||
* [`SetBucketNotification`](https://min.io/docs/minio/linux/developers/go/API.html#SetBucketNotification)
|
||||
* [`GetBucketNotification`](https://min.io/docs/minio/linux/developers/go/API.html#GetBucketNotification)
|
||||
* [`RemoveAllBucketNotification`](https://min.io/docs/minio/linux/developers/go/API.html#RemoveAllBucketNotification)
|
||||
* [`ListenBucketNotification`](https://min.io/docs/minio/linux/developers/go/API.html#ListenBucketNotification) (MinIO Extension)
|
||||
* [`ListenNotification`](https://min.io/docs/minio/linux/developers/go/API.html#ListenNotification) (MinIO Extension)
|
||||
|
||||
### API Reference : File Object Operations
|
||||
|
||||
* [`FPutObject`](https://min.io/docs/minio/linux/developers/go/API.html#FPutObject)
|
||||
* [`FGetObject`](https://min.io/docs/minio/linux/developers/go/API.html#FGetObject)
|
||||
|
||||
### API Reference : Object Operations
|
||||
|
||||
* [`GetObject`](https://min.io/docs/minio/linux/developers/go/API.html#GetObject)
|
||||
* [`PutObject`](https://min.io/docs/minio/linux/developers/go/API.html#PutObject)
|
||||
* [`PutObjectStreaming`](https://min.io/docs/minio/linux/developers/go/API.html#PutObjectStreaming)
|
||||
* [`StatObject`](https://min.io/docs/minio/linux/developers/go/API.html#StatObject)
|
||||
* [`CopyObject`](https://min.io/docs/minio/linux/developers/go/API.html#CopyObject)
|
||||
* [`RemoveObject`](https://min.io/docs/minio/linux/developers/go/API.html#RemoveObject)
|
||||
* [`RemoveObjects`](https://min.io/docs/minio/linux/developers/go/API.html#RemoveObjects)
|
||||
* [`RemoveIncompleteUpload`](https://min.io/docs/minio/linux/developers/go/API.html#RemoveIncompleteUpload)
|
||||
* [`SelectObjectContent`](https://min.io/docs/minio/linux/developers/go/API.html#SelectObjectContent)
|
||||
|
||||
### API Reference : Presigned Operations
|
||||
|
||||
* [`PresignedGetObject`](https://min.io/docs/minio/linux/developers/go/API.html#PresignedGetObject)
|
||||
* [`PresignedPutObject`](https://min.io/docs/minio/linux/developers/go/API.html#PresignedPutObject)
|
||||
* [`PresignedHeadObject`](https://min.io/docs/minio/linux/developers/go/API.html#PresignedHeadObject)
|
||||
* [`PresignedPostPolicy`](https://min.io/docs/minio/linux/developers/go/API.html#PresignedPostPolicy)
|
||||
|
||||
### API Reference : Client custom settings
|
||||
|
||||
* [`SetAppInfo`](https://min.io/docs/minio/linux/developers/go/API.html#SetAppInfo)
|
||||
* [`TraceOn`](https://min.io/docs/minio/linux/developers/go/API.html#TraceOn)
|
||||
* [`TraceOff`](https://min.io/docs/minio/linux/developers/go/API.html#TraceOff)
|
||||
|
||||
## Full Examples
|
||||
|
||||
### Full Examples : Bucket Operations
|
||||
|
||||
* [makebucket.go](https://github.com/minio/minio-go/blob/master/examples/s3/makebucket.go)
|
||||
* [listbuckets.go](https://github.com/minio/minio-go/blob/master/examples/s3/listbuckets.go)
|
||||
* [bucketexists.go](https://github.com/minio/minio-go/blob/master/examples/s3/bucketexists.go)
|
||||
* [removebucket.go](https://github.com/minio/minio-go/blob/master/examples/s3/removebucket.go)
|
||||
* [listobjects.go](https://github.com/minio/minio-go/blob/master/examples/s3/listobjects.go)
|
||||
* [listobjectsV2.go](https://github.com/minio/minio-go/blob/master/examples/s3/listobjectsV2.go)
|
||||
* [listincompleteuploads.go](https://github.com/minio/minio-go/blob/master/examples/s3/listincompleteuploads.go)
|
||||
|
||||
### Full Examples : Bucket policy Operations
|
||||
|
||||
* [setbucketpolicy.go](https://github.com/minio/minio-go/blob/master/examples/s3/setbucketpolicy.go)
|
||||
* [getbucketpolicy.go](https://github.com/minio/minio-go/blob/master/examples/s3/getbucketpolicy.go)
|
||||
* [listbucketpolicies.go](https://github.com/minio/minio-go/blob/master/examples/s3/listbucketpolicies.go)
|
||||
|
||||
### Full Examples : Bucket lifecycle Operations
|
||||
|
||||
* [setbucketlifecycle.go](https://github.com/minio/minio-go/blob/master/examples/s3/setbucketlifecycle.go)
|
||||
* [getbucketlifecycle.go](https://github.com/minio/minio-go/blob/master/examples/s3/getbucketlifecycle.go)
|
||||
|
||||
### Full Examples : Bucket encryption Operations
|
||||
|
||||
* [setbucketencryption.go](https://github.com/minio/minio-go/blob/master/examples/s3/setbucketencryption.go)
|
||||
* [getbucketencryption.go](https://github.com/minio/minio-go/blob/master/examples/s3/getbucketencryption.go)
|
||||
* [removebucketencryption.go](https://github.com/minio/minio-go/blob/master/examples/s3/removebucketencryption.go)
|
||||
|
||||
### Full Examples : Bucket replication Operations
|
||||
|
||||
* [setbucketreplication.go](https://github.com/minio/minio-go/blob/master/examples/s3/setbucketreplication.go)
|
||||
* [getbucketreplication.go](https://github.com/minio/minio-go/blob/master/examples/s3/getbucketreplication.go)
|
||||
* [removebucketreplication.go](https://github.com/minio/minio-go/blob/master/examples/s3/removebucketreplication.go)
|
||||
|
||||
### Full Examples : Bucket notification Operations
|
||||
|
||||
* [setbucketnotification.go](https://github.com/minio/minio-go/blob/master/examples/s3/setbucketnotification.go)
|
||||
* [getbucketnotification.go](https://github.com/minio/minio-go/blob/master/examples/s3/getbucketnotification.go)
|
||||
* [removeallbucketnotification.go](https://github.com/minio/minio-go/blob/master/examples/s3/removeallbucketnotification.go)
|
||||
* [listenbucketnotification.go](https://github.com/minio/minio-go/blob/master/examples/minio/listenbucketnotification.go) (MinIO Extension)
|
||||
* [listennotification.go](https://github.com/minio/minio-go/blob/master/examples/minio/listen-notification.go) (MinIO Extension)
|
||||
|
||||
### Full Examples : File Object Operations
|
||||
|
||||
* [fputobject.go](https://github.com/minio/minio-go/blob/master/examples/s3/fputobject.go)
|
||||
* [fgetobject.go](https://github.com/minio/minio-go/blob/master/examples/s3/fgetobject.go)
|
||||
|
||||
### Full Examples : Object Operations
|
||||
|
||||
* [putobject.go](https://github.com/minio/minio-go/blob/master/examples/s3/putobject.go)
|
||||
* [getobject.go](https://github.com/minio/minio-go/blob/master/examples/s3/getobject.go)
|
||||
* [statobject.go](https://github.com/minio/minio-go/blob/master/examples/s3/statobject.go)
|
||||
* [copyobject.go](https://github.com/minio/minio-go/blob/master/examples/s3/copyobject.go)
|
||||
* [removeobject.go](https://github.com/minio/minio-go/blob/master/examples/s3/removeobject.go)
|
||||
* [removeincompleteupload.go](https://github.com/minio/minio-go/blob/master/examples/s3/removeincompleteupload.go)
|
||||
* [removeobjects.go](https://github.com/minio/minio-go/blob/master/examples/s3/removeobjects.go)
|
||||
|
||||
### Full Examples : Encrypted Object Operations
|
||||
|
||||
* [put-encrypted-object.go](https://github.com/minio/minio-go/blob/master/examples/s3/put-encrypted-object.go)
|
||||
* [get-encrypted-object.go](https://github.com/minio/minio-go/blob/master/examples/s3/get-encrypted-object.go)
|
||||
* [fput-encrypted-object.go](https://github.com/minio/minio-go/blob/master/examples/s3/fputencrypted-object.go)
|
||||
|
||||
### Full Examples : Presigned Operations
|
||||
|
||||
* [presignedgetobject.go](https://github.com/minio/minio-go/blob/master/examples/s3/presignedgetobject.go)
|
||||
* [presignedputobject.go](https://github.com/minio/minio-go/blob/master/examples/s3/presignedputobject.go)
|
||||
* [presignedheadobject.go](https://github.com/minio/minio-go/blob/master/examples/s3/presignedheadobject.go)
|
||||
* [presignedpostpolicy.go](https://github.com/minio/minio-go/blob/master/examples/s3/presignedpostpolicy.go)
|
||||
|
||||
## Explore Further
|
||||
|
||||
* [Godoc Documentation](https://pkg.go.dev/github.com/minio/minio-go/v7)
|
||||
* [Complete Documentation](https://min.io/docs/minio/kubernetes/upstream/index.html)
|
||||
* [MinIO Go Client SDK API Reference](https://min.io/docs/minio/linux/developers/go/API.html)
|
||||
|
||||
## Contribute
|
||||
|
||||
[Contributors Guide](https://github.com/minio/minio-go/blob/master/CONTRIBUTING.md)
|
||||
|
||||
## License
|
||||
|
||||
This SDK is distributed under the [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0), see [LICENSE](https://github.com/minio/minio-go/blob/master/LICENSE) and [NOTICE](https://github.com/minio/minio-go/blob/master/NOTICE) for more information.
|
||||
+136
@@ -0,0 +1,136 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2024 MinIO, Inc.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"net/http"
|
||||
"net/url"
|
||||
|
||||
"github.com/minio/minio-go/v7/pkg/cors"
|
||||
"github.com/minio/minio-go/v7/pkg/s3utils"
|
||||
)
|
||||
|
||||
// SetBucketCors sets the cors configuration for the bucket
|
||||
func (c *Client) SetBucketCors(ctx context.Context, bucketName string, corsConfig *cors.Config) error {
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if corsConfig == nil {
|
||||
return c.removeBucketCors(ctx, bucketName)
|
||||
}
|
||||
|
||||
return c.putBucketCors(ctx, bucketName, corsConfig)
|
||||
}
|
||||
|
||||
func (c *Client) putBucketCors(ctx context.Context, bucketName string, corsConfig *cors.Config) error {
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("cors", "")
|
||||
|
||||
corsStr, err := corsConfig.ToXML()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
reqMetadata := requestMetadata{
|
||||
bucketName: bucketName,
|
||||
queryValues: urlValues,
|
||||
contentBody: bytes.NewReader(corsStr),
|
||||
contentLength: int64(len(corsStr)),
|
||||
contentMD5Base64: sumMD5Base64([]byte(corsStr)),
|
||||
}
|
||||
|
||||
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if resp != nil {
|
||||
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusNoContent {
|
||||
return httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *Client) removeBucketCors(ctx context.Context, bucketName string) error {
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("cors", "")
|
||||
|
||||
resp, err := c.executeMethod(ctx, http.MethodDelete, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
queryValues: urlValues,
|
||||
contentSHA256Hex: emptySHA256Hex,
|
||||
})
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if resp.StatusCode != http.StatusNoContent {
|
||||
return httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetBucketCors returns the current cors
|
||||
func (c *Client) GetBucketCors(ctx context.Context, bucketName string) (*cors.Config, error) {
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
bucketCors, err := c.getBucketCors(ctx, bucketName)
|
||||
if err != nil {
|
||||
errResponse := ToErrorResponse(err)
|
||||
if errResponse.Code == "NoSuchCORSConfiguration" {
|
||||
return nil, nil
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
return bucketCors, nil
|
||||
}
|
||||
|
||||
func (c *Client) getBucketCors(ctx context.Context, bucketName string) (*cors.Config, error) {
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("cors", "")
|
||||
|
||||
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
queryValues: urlValues,
|
||||
contentSHA256Hex: emptySHA256Hex, // TODO: needed? copied over from other example, but not spec'd in API.
|
||||
})
|
||||
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if resp != nil {
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return nil, httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
}
|
||||
|
||||
corsConfig, err := cors.ParseBucketCorsConfig(resp.Body)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return corsConfig, nil
|
||||
}
|
||||
+134
@@ -0,0 +1,134 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2020 MinIO, Inc.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/xml"
|
||||
"net/http"
|
||||
"net/url"
|
||||
|
||||
"github.com/minio/minio-go/v7/pkg/s3utils"
|
||||
"github.com/minio/minio-go/v7/pkg/sse"
|
||||
)
|
||||
|
||||
// SetBucketEncryption sets the default encryption configuration on an existing bucket.
|
||||
func (c *Client) SetBucketEncryption(ctx context.Context, bucketName string, config *sse.Configuration) error {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if config == nil {
|
||||
return errInvalidArgument("configuration cannot be empty")
|
||||
}
|
||||
|
||||
buf, err := xml.Marshal(config)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Get resources properly escaped and lined up before
|
||||
// using them in http request.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("encryption", "")
|
||||
|
||||
// Content-length is mandatory to set a default encryption configuration
|
||||
reqMetadata := requestMetadata{
|
||||
bucketName: bucketName,
|
||||
queryValues: urlValues,
|
||||
contentBody: bytes.NewReader(buf),
|
||||
contentLength: int64(len(buf)),
|
||||
contentMD5Base64: sumMD5Base64(buf),
|
||||
}
|
||||
|
||||
// Execute PUT to upload a new bucket default encryption configuration.
|
||||
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// RemoveBucketEncryption removes the default encryption configuration on a bucket with a context to control cancellations and timeouts.
|
||||
func (c *Client) RemoveBucketEncryption(ctx context.Context, bucketName string) error {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Get resources properly escaped and lined up before
|
||||
// using them in http request.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("encryption", "")
|
||||
|
||||
// DELETE default encryption configuration on a bucket.
|
||||
resp, err := c.executeMethod(ctx, http.MethodDelete, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
queryValues: urlValues,
|
||||
contentSHA256Hex: emptySHA256Hex,
|
||||
})
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusNoContent {
|
||||
return httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetBucketEncryption gets the default encryption configuration
|
||||
// on an existing bucket with a context to control cancellations and timeouts.
|
||||
func (c *Client) GetBucketEncryption(ctx context.Context, bucketName string) (*sse.Configuration, error) {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Get resources properly escaped and lined up before
|
||||
// using them in http request.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("encryption", "")
|
||||
|
||||
// Execute GET on bucket to get the default encryption configuration.
|
||||
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
queryValues: urlValues,
|
||||
})
|
||||
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return nil, httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
|
||||
encryptionConfig := &sse.Configuration{}
|
||||
if err = xmlDecoder(resp.Body, encryptionConfig); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return encryptionConfig, nil
|
||||
}
|
||||
+169
@@ -0,0 +1,169 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2020 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/xml"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"time"
|
||||
|
||||
"github.com/minio/minio-go/v7/pkg/lifecycle"
|
||||
"github.com/minio/minio-go/v7/pkg/s3utils"
|
||||
)
|
||||
|
||||
// SetBucketLifecycle set the lifecycle on an existing bucket.
|
||||
func (c *Client) SetBucketLifecycle(ctx context.Context, bucketName string, config *lifecycle.Configuration) error {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// If lifecycle is empty then delete it.
|
||||
if config.Empty() {
|
||||
return c.removeBucketLifecycle(ctx, bucketName)
|
||||
}
|
||||
|
||||
buf, err := xml.Marshal(config)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Save the updated lifecycle.
|
||||
return c.putBucketLifecycle(ctx, bucketName, buf)
|
||||
}
|
||||
|
||||
// Saves a new bucket lifecycle.
|
||||
func (c *Client) putBucketLifecycle(ctx context.Context, bucketName string, buf []byte) error {
|
||||
// Get resources properly escaped and lined up before
|
||||
// using them in http request.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("lifecycle", "")
|
||||
|
||||
// Content-length is mandatory for put lifecycle request
|
||||
reqMetadata := requestMetadata{
|
||||
bucketName: bucketName,
|
||||
queryValues: urlValues,
|
||||
contentBody: bytes.NewReader(buf),
|
||||
contentLength: int64(len(buf)),
|
||||
contentMD5Base64: sumMD5Base64(buf),
|
||||
}
|
||||
|
||||
// Execute PUT to upload a new bucket lifecycle.
|
||||
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if resp != nil {
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Remove lifecycle from a bucket.
|
||||
func (c *Client) removeBucketLifecycle(ctx context.Context, bucketName string) error {
|
||||
// Get resources properly escaped and lined up before
|
||||
// using them in http request.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("lifecycle", "")
|
||||
|
||||
// Execute DELETE on objectName.
|
||||
resp, err := c.executeMethod(ctx, http.MethodDelete, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
queryValues: urlValues,
|
||||
contentSHA256Hex: emptySHA256Hex,
|
||||
})
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetBucketLifecycle fetch bucket lifecycle configuration
|
||||
func (c *Client) GetBucketLifecycle(ctx context.Context, bucketName string) (*lifecycle.Configuration, error) {
|
||||
lc, _, err := c.GetBucketLifecycleWithInfo(ctx, bucketName)
|
||||
return lc, err
|
||||
}
|
||||
|
||||
// GetBucketLifecycleWithInfo fetch bucket lifecycle configuration along with when it was last updated
|
||||
func (c *Client) GetBucketLifecycleWithInfo(ctx context.Context, bucketName string) (*lifecycle.Configuration, time.Time, error) {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return nil, time.Time{}, err
|
||||
}
|
||||
|
||||
bucketLifecycle, updatedAt, err := c.getBucketLifecycle(ctx, bucketName)
|
||||
if err != nil {
|
||||
return nil, time.Time{}, err
|
||||
}
|
||||
|
||||
config := lifecycle.NewConfiguration()
|
||||
if err = xml.Unmarshal(bucketLifecycle, config); err != nil {
|
||||
return nil, time.Time{}, err
|
||||
}
|
||||
return config, updatedAt, nil
|
||||
}
|
||||
|
||||
// Request server for current bucket lifecycle.
|
||||
func (c *Client) getBucketLifecycle(ctx context.Context, bucketName string) ([]byte, time.Time, error) {
|
||||
// Get resources properly escaped and lined up before
|
||||
// using them in http request.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("lifecycle", "")
|
||||
urlValues.Set("withUpdatedAt", "true")
|
||||
|
||||
// Execute GET on bucket to get lifecycle.
|
||||
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
queryValues: urlValues,
|
||||
})
|
||||
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return nil, time.Time{}, err
|
||||
}
|
||||
|
||||
if resp != nil {
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return nil, time.Time{}, httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
}
|
||||
|
||||
lcBytes, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
return nil, time.Time{}, err
|
||||
}
|
||||
|
||||
const minIOLifecycleCfgUpdatedAt = "X-Minio-LifecycleConfig-UpdatedAt"
|
||||
var updatedAt time.Time
|
||||
if timeStr := resp.Header.Get(minIOLifecycleCfgUpdatedAt); timeStr != "" {
|
||||
updatedAt, err = time.Parse(iso8601DateFormat, timeStr)
|
||||
if err != nil {
|
||||
return nil, time.Time{}, err
|
||||
}
|
||||
}
|
||||
|
||||
return lcBytes, updatedAt, nil
|
||||
}
|
||||
+260
@@ -0,0 +1,260 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2017-2020 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/xml"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"time"
|
||||
|
||||
"github.com/goccy/go-json"
|
||||
"github.com/minio/minio-go/v7/pkg/notification"
|
||||
"github.com/minio/minio-go/v7/pkg/s3utils"
|
||||
)
|
||||
|
||||
// SetBucketNotification saves a new bucket notification with a context to control cancellations and timeouts.
|
||||
func (c *Client) SetBucketNotification(ctx context.Context, bucketName string, config notification.Configuration) error {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Get resources properly escaped and lined up before
|
||||
// using them in http request.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("notification", "")
|
||||
|
||||
notifBytes, err := xml.Marshal(&config)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
notifBuffer := bytes.NewReader(notifBytes)
|
||||
reqMetadata := requestMetadata{
|
||||
bucketName: bucketName,
|
||||
queryValues: urlValues,
|
||||
contentBody: notifBuffer,
|
||||
contentLength: int64(len(notifBytes)),
|
||||
contentMD5Base64: sumMD5Base64(notifBytes),
|
||||
contentSHA256Hex: sum256Hex(notifBytes),
|
||||
}
|
||||
|
||||
// Execute PUT to upload a new bucket notification.
|
||||
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if resp != nil {
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// RemoveAllBucketNotification - Remove bucket notification clears all previously specified config
|
||||
func (c *Client) RemoveAllBucketNotification(ctx context.Context, bucketName string) error {
|
||||
return c.SetBucketNotification(ctx, bucketName, notification.Configuration{})
|
||||
}
|
||||
|
||||
// GetBucketNotification returns current bucket notification configuration
|
||||
func (c *Client) GetBucketNotification(ctx context.Context, bucketName string) (bucketNotification notification.Configuration, err error) {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return notification.Configuration{}, err
|
||||
}
|
||||
return c.getBucketNotification(ctx, bucketName)
|
||||
}
|
||||
|
||||
// Request server for notification rules.
|
||||
func (c *Client) getBucketNotification(ctx context.Context, bucketName string) (notification.Configuration, error) {
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("notification", "")
|
||||
|
||||
// Execute GET on bucket to list objects.
|
||||
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
queryValues: urlValues,
|
||||
contentSHA256Hex: emptySHA256Hex,
|
||||
})
|
||||
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return notification.Configuration{}, err
|
||||
}
|
||||
return processBucketNotificationResponse(bucketName, resp)
|
||||
}
|
||||
|
||||
// processes the GetNotification http response from the server.
|
||||
func processBucketNotificationResponse(bucketName string, resp *http.Response) (notification.Configuration, error) {
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
errResponse := httpRespToErrorResponse(resp, bucketName, "")
|
||||
return notification.Configuration{}, errResponse
|
||||
}
|
||||
var bucketNotification notification.Configuration
|
||||
err := xmlDecoder(resp.Body, &bucketNotification)
|
||||
if err != nil {
|
||||
return notification.Configuration{}, err
|
||||
}
|
||||
return bucketNotification, nil
|
||||
}
|
||||
|
||||
// ListenNotification listen for all events, this is a MinIO specific API
|
||||
func (c *Client) ListenNotification(ctx context.Context, prefix, suffix string, events []string) <-chan notification.Info {
|
||||
return c.ListenBucketNotification(ctx, "", prefix, suffix, events)
|
||||
}
|
||||
|
||||
// ListenBucketNotification listen for bucket events, this is a MinIO specific API
|
||||
func (c *Client) ListenBucketNotification(ctx context.Context, bucketName, prefix, suffix string, events []string) <-chan notification.Info {
|
||||
notificationInfoCh := make(chan notification.Info, 1)
|
||||
const notificationCapacity = 4 * 1024 * 1024
|
||||
notificationEventBuffer := make([]byte, notificationCapacity)
|
||||
// Only success, start a routine to start reading line by line.
|
||||
go func(notificationInfoCh chan<- notification.Info) {
|
||||
defer close(notificationInfoCh)
|
||||
|
||||
// Validate the bucket name.
|
||||
if bucketName != "" {
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
select {
|
||||
case notificationInfoCh <- notification.Info{
|
||||
Err: err,
|
||||
}:
|
||||
case <-ctx.Done():
|
||||
}
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// Check ARN partition to verify if listening bucket is supported
|
||||
if s3utils.IsAmazonEndpoint(*c.endpointURL) || s3utils.IsGoogleEndpoint(*c.endpointURL) {
|
||||
select {
|
||||
case notificationInfoCh <- notification.Info{
|
||||
Err: errAPINotSupported("Listening for bucket notification is specific only to `minio` server endpoints"),
|
||||
}:
|
||||
case <-ctx.Done():
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// Continuously run and listen on bucket notification.
|
||||
// Create a done channel to control 'ListObjects' go routine.
|
||||
retryDoneCh := make(chan struct{}, 1)
|
||||
|
||||
// Indicate to our routine to exit cleanly upon return.
|
||||
defer close(retryDoneCh)
|
||||
|
||||
// Prepare urlValues to pass into the request on every loop
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("ping", "10")
|
||||
urlValues.Set("prefix", prefix)
|
||||
urlValues.Set("suffix", suffix)
|
||||
urlValues["events"] = events
|
||||
|
||||
// Wait on the jitter retry loop.
|
||||
for range c.newRetryTimerContinous(time.Second, time.Second*30, MaxJitter, retryDoneCh) {
|
||||
// Execute GET on bucket to list objects.
|
||||
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
queryValues: urlValues,
|
||||
contentSHA256Hex: emptySHA256Hex,
|
||||
})
|
||||
if err != nil {
|
||||
select {
|
||||
case notificationInfoCh <- notification.Info{
|
||||
Err: err,
|
||||
}:
|
||||
case <-ctx.Done():
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// Validate http response, upon error return quickly.
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
errResponse := httpRespToErrorResponse(resp, bucketName, "")
|
||||
select {
|
||||
case notificationInfoCh <- notification.Info{
|
||||
Err: errResponse,
|
||||
}:
|
||||
case <-ctx.Done():
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// Initialize a new bufio scanner, to read line by line.
|
||||
bio := bufio.NewScanner(resp.Body)
|
||||
|
||||
// Use a higher buffer to support unexpected
|
||||
// caching done by proxies
|
||||
bio.Buffer(notificationEventBuffer, notificationCapacity)
|
||||
|
||||
// Unmarshal each line, returns marshaled values.
|
||||
for bio.Scan() {
|
||||
var notificationInfo notification.Info
|
||||
if err = json.Unmarshal(bio.Bytes(), ¬ificationInfo); err != nil {
|
||||
// Unexpected error during json unmarshal, send
|
||||
// the error to caller for actionable as needed.
|
||||
select {
|
||||
case notificationInfoCh <- notification.Info{
|
||||
Err: err,
|
||||
}:
|
||||
case <-ctx.Done():
|
||||
return
|
||||
}
|
||||
closeResponse(resp)
|
||||
continue
|
||||
}
|
||||
|
||||
// Empty events pinged from the server
|
||||
if len(notificationInfo.Records) == 0 && notificationInfo.Err == nil {
|
||||
continue
|
||||
}
|
||||
|
||||
// Send notificationInfo
|
||||
select {
|
||||
case notificationInfoCh <- notificationInfo:
|
||||
case <-ctx.Done():
|
||||
closeResponse(resp)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
if err = bio.Err(); err != nil {
|
||||
select {
|
||||
case notificationInfoCh <- notification.Info{
|
||||
Err: err,
|
||||
}:
|
||||
case <-ctx.Done():
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// Close current connection before looping further.
|
||||
closeResponse(resp)
|
||||
|
||||
}
|
||||
}(notificationInfoCh)
|
||||
|
||||
// Returns the notification info channel, for caller to start reading from.
|
||||
return notificationInfoCh
|
||||
}
|
||||
+147
@@ -0,0 +1,147 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2020 MinIO, Inc.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
|
||||
"github.com/minio/minio-go/v7/pkg/s3utils"
|
||||
)
|
||||
|
||||
// SetBucketPolicy sets the access permissions on an existing bucket.
|
||||
func (c *Client) SetBucketPolicy(ctx context.Context, bucketName, policy string) error {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// If policy is empty then delete the bucket policy.
|
||||
if policy == "" {
|
||||
return c.removeBucketPolicy(ctx, bucketName)
|
||||
}
|
||||
|
||||
// Save the updated policies.
|
||||
return c.putBucketPolicy(ctx, bucketName, policy)
|
||||
}
|
||||
|
||||
// Saves a new bucket policy.
|
||||
func (c *Client) putBucketPolicy(ctx context.Context, bucketName, policy string) error {
|
||||
// Get resources properly escaped and lined up before
|
||||
// using them in http request.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("policy", "")
|
||||
|
||||
reqMetadata := requestMetadata{
|
||||
bucketName: bucketName,
|
||||
queryValues: urlValues,
|
||||
contentBody: strings.NewReader(policy),
|
||||
contentLength: int64(len(policy)),
|
||||
}
|
||||
|
||||
// Execute PUT to upload a new bucket policy.
|
||||
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if resp != nil {
|
||||
if resp.StatusCode != http.StatusNoContent && resp.StatusCode != http.StatusOK {
|
||||
return httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Removes all policies on a bucket.
|
||||
func (c *Client) removeBucketPolicy(ctx context.Context, bucketName string) error {
|
||||
// Get resources properly escaped and lined up before
|
||||
// using them in http request.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("policy", "")
|
||||
|
||||
// Execute DELETE on objectName.
|
||||
resp, err := c.executeMethod(ctx, http.MethodDelete, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
queryValues: urlValues,
|
||||
contentSHA256Hex: emptySHA256Hex,
|
||||
})
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if resp.StatusCode != http.StatusNoContent {
|
||||
return httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetBucketPolicy returns the current policy
|
||||
func (c *Client) GetBucketPolicy(ctx context.Context, bucketName string) (string, error) {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return "", err
|
||||
}
|
||||
bucketPolicy, err := c.getBucketPolicy(ctx, bucketName)
|
||||
if err != nil {
|
||||
errResponse := ToErrorResponse(err)
|
||||
if errResponse.Code == "NoSuchBucketPolicy" {
|
||||
return "", nil
|
||||
}
|
||||
return "", err
|
||||
}
|
||||
return bucketPolicy, nil
|
||||
}
|
||||
|
||||
// Request server for current bucket policy.
|
||||
func (c *Client) getBucketPolicy(ctx context.Context, bucketName string) (string, error) {
|
||||
// Get resources properly escaped and lined up before
|
||||
// using them in http request.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("policy", "")
|
||||
|
||||
// Execute GET on bucket to list objects.
|
||||
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
queryValues: urlValues,
|
||||
contentSHA256Hex: emptySHA256Hex,
|
||||
})
|
||||
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
if resp != nil {
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return "", httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
}
|
||||
|
||||
bucketPolicyBuf, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
policy := string(bucketPolicyBuf)
|
||||
return policy, err
|
||||
}
|
||||
+355
@@ -0,0 +1,355 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2020 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"encoding/xml"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/minio/minio-go/v7/pkg/replication"
|
||||
"github.com/minio/minio-go/v7/pkg/s3utils"
|
||||
)
|
||||
|
||||
// RemoveBucketReplication removes a replication config on an existing bucket.
|
||||
func (c *Client) RemoveBucketReplication(ctx context.Context, bucketName string) error {
|
||||
return c.removeBucketReplication(ctx, bucketName)
|
||||
}
|
||||
|
||||
// SetBucketReplication sets a replication config on an existing bucket.
|
||||
func (c *Client) SetBucketReplication(ctx context.Context, bucketName string, cfg replication.Config) error {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// If replication is empty then delete it.
|
||||
if cfg.Empty() {
|
||||
return c.removeBucketReplication(ctx, bucketName)
|
||||
}
|
||||
// Save the updated replication.
|
||||
return c.putBucketReplication(ctx, bucketName, cfg)
|
||||
}
|
||||
|
||||
// Saves a new bucket replication.
|
||||
func (c *Client) putBucketReplication(ctx context.Context, bucketName string, cfg replication.Config) error {
|
||||
// Get resources properly escaped and lined up before
|
||||
// using them in http request.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("replication", "")
|
||||
replication, err := xml.Marshal(cfg)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
reqMetadata := requestMetadata{
|
||||
bucketName: bucketName,
|
||||
queryValues: urlValues,
|
||||
contentBody: bytes.NewReader(replication),
|
||||
contentLength: int64(len(replication)),
|
||||
contentMD5Base64: sumMD5Base64(replication),
|
||||
}
|
||||
|
||||
// Execute PUT to upload a new bucket replication config.
|
||||
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// Remove replication from a bucket.
|
||||
func (c *Client) removeBucketReplication(ctx context.Context, bucketName string) error {
|
||||
// Get resources properly escaped and lined up before
|
||||
// using them in http request.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("replication", "")
|
||||
|
||||
// Execute DELETE on objectName.
|
||||
resp, err := c.executeMethod(ctx, http.MethodDelete, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
queryValues: urlValues,
|
||||
contentSHA256Hex: emptySHA256Hex,
|
||||
})
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetBucketReplication fetches bucket replication configuration.If config is not
|
||||
// found, returns empty config with nil error.
|
||||
func (c *Client) GetBucketReplication(ctx context.Context, bucketName string) (cfg replication.Config, err error) {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return cfg, err
|
||||
}
|
||||
bucketReplicationCfg, err := c.getBucketReplication(ctx, bucketName)
|
||||
if err != nil {
|
||||
errResponse := ToErrorResponse(err)
|
||||
if errResponse.Code == "ReplicationConfigurationNotFoundError" {
|
||||
return cfg, nil
|
||||
}
|
||||
return cfg, err
|
||||
}
|
||||
return bucketReplicationCfg, nil
|
||||
}
|
||||
|
||||
// Request server for current bucket replication config.
|
||||
func (c *Client) getBucketReplication(ctx context.Context, bucketName string) (cfg replication.Config, err error) {
|
||||
// Get resources properly escaped and lined up before
|
||||
// using them in http request.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("replication", "")
|
||||
|
||||
// Execute GET on bucket to get replication config.
|
||||
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
queryValues: urlValues,
|
||||
})
|
||||
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return cfg, err
|
||||
}
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return cfg, httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
|
||||
if err = xmlDecoder(resp.Body, &cfg); err != nil {
|
||||
return cfg, err
|
||||
}
|
||||
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
// GetBucketReplicationMetrics fetches bucket replication status metrics
|
||||
func (c *Client) GetBucketReplicationMetrics(ctx context.Context, bucketName string) (s replication.Metrics, err error) {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return s, err
|
||||
}
|
||||
// Get resources properly escaped and lined up before
|
||||
// using them in http request.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("replication-metrics", "")
|
||||
|
||||
// Execute GET on bucket to get replication config.
|
||||
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
queryValues: urlValues,
|
||||
})
|
||||
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return s, err
|
||||
}
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return s, httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
respBytes, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
return s, err
|
||||
}
|
||||
|
||||
if err := json.Unmarshal(respBytes, &s); err != nil {
|
||||
return s, err
|
||||
}
|
||||
return s, nil
|
||||
}
|
||||
|
||||
// mustGetUUID - get a random UUID.
|
||||
func mustGetUUID() string {
|
||||
u, err := uuid.NewRandom()
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return u.String()
|
||||
}
|
||||
|
||||
// ResetBucketReplication kicks off replication of previously replicated objects if ExistingObjectReplication
|
||||
// is enabled in the replication config
|
||||
func (c *Client) ResetBucketReplication(ctx context.Context, bucketName string, olderThan time.Duration) (rID string, err error) {
|
||||
rID = mustGetUUID()
|
||||
_, err = c.resetBucketReplicationOnTarget(ctx, bucketName, olderThan, "", rID)
|
||||
if err != nil {
|
||||
return rID, err
|
||||
}
|
||||
return rID, nil
|
||||
}
|
||||
|
||||
// ResetBucketReplicationOnTarget kicks off replication of previously replicated objects if
|
||||
// ExistingObjectReplication is enabled in the replication config
|
||||
func (c *Client) ResetBucketReplicationOnTarget(ctx context.Context, bucketName string, olderThan time.Duration, tgtArn string) (replication.ResyncTargetsInfo, error) {
|
||||
return c.resetBucketReplicationOnTarget(ctx, bucketName, olderThan, tgtArn, mustGetUUID())
|
||||
}
|
||||
|
||||
// ResetBucketReplication kicks off replication of previously replicated objects if ExistingObjectReplication
|
||||
// is enabled in the replication config
|
||||
func (c *Client) resetBucketReplicationOnTarget(ctx context.Context, bucketName string, olderThan time.Duration, tgtArn, resetID string) (rinfo replication.ResyncTargetsInfo, err error) {
|
||||
// Input validation.
|
||||
if err = s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return
|
||||
}
|
||||
// Get resources properly escaped and lined up before
|
||||
// using them in http request.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("replication-reset", "")
|
||||
if olderThan > 0 {
|
||||
urlValues.Set("older-than", olderThan.String())
|
||||
}
|
||||
if tgtArn != "" {
|
||||
urlValues.Set("arn", tgtArn)
|
||||
}
|
||||
urlValues.Set("reset-id", resetID)
|
||||
// Execute GET on bucket to get replication config.
|
||||
resp, err := c.executeMethod(ctx, http.MethodPut, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
queryValues: urlValues,
|
||||
})
|
||||
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return rinfo, err
|
||||
}
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return rinfo, httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
|
||||
if err = json.NewDecoder(resp.Body).Decode(&rinfo); err != nil {
|
||||
return rinfo, err
|
||||
}
|
||||
return rinfo, nil
|
||||
}
|
||||
|
||||
// GetBucketReplicationResyncStatus gets the status of replication resync
|
||||
func (c *Client) GetBucketReplicationResyncStatus(ctx context.Context, bucketName, arn string) (rinfo replication.ResyncTargetsInfo, err error) {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return rinfo, err
|
||||
}
|
||||
// Get resources properly escaped and lined up before
|
||||
// using them in http request.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("replication-reset-status", "")
|
||||
if arn != "" {
|
||||
urlValues.Set("arn", arn)
|
||||
}
|
||||
// Execute GET on bucket to get replication config.
|
||||
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
queryValues: urlValues,
|
||||
})
|
||||
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return rinfo, err
|
||||
}
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return rinfo, httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
|
||||
if err = json.NewDecoder(resp.Body).Decode(&rinfo); err != nil {
|
||||
return rinfo, err
|
||||
}
|
||||
return rinfo, nil
|
||||
}
|
||||
|
||||
// GetBucketReplicationMetricsV2 fetches bucket replication status metrics
|
||||
func (c *Client) GetBucketReplicationMetricsV2(ctx context.Context, bucketName string) (s replication.MetricsV2, err error) {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return s, err
|
||||
}
|
||||
// Get resources properly escaped and lined up before
|
||||
// using them in http request.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("replication-metrics", "2")
|
||||
|
||||
// Execute GET on bucket to get replication metrics.
|
||||
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
queryValues: urlValues,
|
||||
})
|
||||
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return s, err
|
||||
}
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return s, httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
respBytes, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
return s, err
|
||||
}
|
||||
|
||||
if err := json.Unmarshal(respBytes, &s); err != nil {
|
||||
return s, err
|
||||
}
|
||||
return s, nil
|
||||
}
|
||||
|
||||
// CheckBucketReplication validates if replication is set up properly for a bucket
|
||||
func (c *Client) CheckBucketReplication(ctx context.Context, bucketName string) (err error) {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return err
|
||||
}
|
||||
// Get resources properly escaped and lined up before
|
||||
// using them in http request.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("replication-check", "")
|
||||
|
||||
// Execute GET on bucket to get replication config.
|
||||
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
queryValues: urlValues,
|
||||
})
|
||||
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
+134
@@ -0,0 +1,134 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2020 MinIO, Inc.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/xml"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
|
||||
"github.com/minio/minio-go/v7/pkg/s3utils"
|
||||
"github.com/minio/minio-go/v7/pkg/tags"
|
||||
)
|
||||
|
||||
// GetBucketTagging fetch tagging configuration for a bucket with a
|
||||
// context to control cancellations and timeouts.
|
||||
func (c *Client) GetBucketTagging(ctx context.Context, bucketName string) (*tags.Tags, error) {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Get resources properly escaped and lined up before
|
||||
// using them in http request.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("tagging", "")
|
||||
|
||||
// Execute GET on bucket to get tagging configuration.
|
||||
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
queryValues: urlValues,
|
||||
})
|
||||
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return nil, httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
|
||||
defer io.Copy(io.Discard, resp.Body)
|
||||
return tags.ParseBucketXML(resp.Body)
|
||||
}
|
||||
|
||||
// SetBucketTagging sets tagging configuration for a bucket
|
||||
// with a context to control cancellations and timeouts.
|
||||
func (c *Client) SetBucketTagging(ctx context.Context, bucketName string, tags *tags.Tags) error {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if tags == nil {
|
||||
return errors.New("nil tags passed")
|
||||
}
|
||||
|
||||
buf, err := xml.Marshal(tags)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Get resources properly escaped and lined up before
|
||||
// using them in http request.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("tagging", "")
|
||||
|
||||
// Content-length is mandatory to set a default encryption configuration
|
||||
reqMetadata := requestMetadata{
|
||||
bucketName: bucketName,
|
||||
queryValues: urlValues,
|
||||
contentBody: bytes.NewReader(buf),
|
||||
contentLength: int64(len(buf)),
|
||||
contentMD5Base64: sumMD5Base64(buf),
|
||||
}
|
||||
|
||||
// Execute PUT on bucket to put tagging configuration.
|
||||
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusNoContent {
|
||||
return httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// RemoveBucketTagging removes tagging configuration for a
|
||||
// bucket with a context to control cancellations and timeouts.
|
||||
func (c *Client) RemoveBucketTagging(ctx context.Context, bucketName string) error {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Get resources properly escaped and lined up before
|
||||
// using them in http request.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("tagging", "")
|
||||
|
||||
// Execute DELETE on bucket to remove tagging configuration.
|
||||
resp, err := c.executeMethod(ctx, http.MethodDelete, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
queryValues: urlValues,
|
||||
contentSHA256Hex: emptySHA256Hex,
|
||||
})
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusNoContent {
|
||||
return httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
+146
@@ -0,0 +1,146 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2020 MinIO, Inc.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/xml"
|
||||
"net/http"
|
||||
"net/url"
|
||||
|
||||
"github.com/minio/minio-go/v7/pkg/s3utils"
|
||||
)
|
||||
|
||||
// SetBucketVersioning sets a bucket versioning configuration
|
||||
func (c *Client) SetBucketVersioning(ctx context.Context, bucketName string, config BucketVersioningConfiguration) error {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
buf, err := xml.Marshal(config)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Get resources properly escaped and lined up before
|
||||
// using them in http request.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("versioning", "")
|
||||
|
||||
reqMetadata := requestMetadata{
|
||||
bucketName: bucketName,
|
||||
queryValues: urlValues,
|
||||
contentBody: bytes.NewReader(buf),
|
||||
contentLength: int64(len(buf)),
|
||||
contentMD5Base64: sumMD5Base64(buf),
|
||||
contentSHA256Hex: sum256Hex(buf),
|
||||
}
|
||||
|
||||
// Execute PUT to set a bucket versioning.
|
||||
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if resp != nil {
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// EnableVersioning - enable object versioning in given bucket.
|
||||
func (c *Client) EnableVersioning(ctx context.Context, bucketName string) error {
|
||||
return c.SetBucketVersioning(ctx, bucketName, BucketVersioningConfiguration{Status: "Enabled"})
|
||||
}
|
||||
|
||||
// SuspendVersioning - suspend object versioning in given bucket.
|
||||
func (c *Client) SuspendVersioning(ctx context.Context, bucketName string) error {
|
||||
return c.SetBucketVersioning(ctx, bucketName, BucketVersioningConfiguration{Status: "Suspended"})
|
||||
}
|
||||
|
||||
// ExcludedPrefix - holds individual prefixes excluded from being versioned.
|
||||
type ExcludedPrefix struct {
|
||||
Prefix string
|
||||
}
|
||||
|
||||
// BucketVersioningConfiguration is the versioning configuration structure
|
||||
type BucketVersioningConfiguration struct {
|
||||
XMLName xml.Name `xml:"VersioningConfiguration"`
|
||||
Status string `xml:"Status"`
|
||||
MFADelete string `xml:"MfaDelete,omitempty"`
|
||||
// MinIO extension - allows selective, prefix-level versioning exclusion.
|
||||
// Requires versioning to be enabled
|
||||
ExcludedPrefixes []ExcludedPrefix `xml:",omitempty"`
|
||||
ExcludeFolders bool `xml:",omitempty"`
|
||||
}
|
||||
|
||||
// Various supported states
|
||||
const (
|
||||
Enabled = "Enabled"
|
||||
// Disabled State = "Disabled" only used by MFA Delete not supported yet.
|
||||
Suspended = "Suspended"
|
||||
)
|
||||
|
||||
// Enabled returns true if bucket versioning is enabled
|
||||
func (b BucketVersioningConfiguration) Enabled() bool {
|
||||
return b.Status == Enabled
|
||||
}
|
||||
|
||||
// Suspended returns true if bucket versioning is suspended
|
||||
func (b BucketVersioningConfiguration) Suspended() bool {
|
||||
return b.Status == Suspended
|
||||
}
|
||||
|
||||
// GetBucketVersioning gets the versioning configuration on
|
||||
// an existing bucket with a context to control cancellations and timeouts.
|
||||
func (c *Client) GetBucketVersioning(ctx context.Context, bucketName string) (BucketVersioningConfiguration, error) {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return BucketVersioningConfiguration{}, err
|
||||
}
|
||||
|
||||
// Get resources properly escaped and lined up before
|
||||
// using them in http request.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("versioning", "")
|
||||
|
||||
// Execute GET on bucket to get the versioning configuration.
|
||||
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
queryValues: urlValues,
|
||||
})
|
||||
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return BucketVersioningConfiguration{}, err
|
||||
}
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return BucketVersioningConfiguration{}, httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
|
||||
versioningConfig := BucketVersioningConfiguration{}
|
||||
if err = xmlDecoder(resp.Body, &versioningConfig); err != nil {
|
||||
return versioningConfig, err
|
||||
}
|
||||
|
||||
return versioningConfig, nil
|
||||
}
|
||||
+594
@@ -0,0 +1,594 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2017, 2018 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/minio/minio-go/v7/pkg/encrypt"
|
||||
"github.com/minio/minio-go/v7/pkg/s3utils"
|
||||
)
|
||||
|
||||
// CopyDestOptions represents options specified by user for CopyObject/ComposeObject APIs
|
||||
type CopyDestOptions struct {
|
||||
Bucket string // points to destination bucket
|
||||
Object string // points to destination object
|
||||
|
||||
// `Encryption` is the key info for server-side-encryption with customer
|
||||
// provided key. If it is nil, no encryption is performed.
|
||||
Encryption encrypt.ServerSide
|
||||
|
||||
// `userMeta` is the user-metadata key-value pairs to be set on the
|
||||
// destination. The keys are automatically prefixed with `x-amz-meta-`
|
||||
// if needed. If nil is passed, and if only a single source (of any
|
||||
// size) is provided in the ComposeObject call, then metadata from the
|
||||
// source is copied to the destination.
|
||||
// if no user-metadata is provided, it is copied from source
|
||||
// (when there is only once source object in the compose
|
||||
// request)
|
||||
UserMetadata map[string]string
|
||||
// UserMetadata is only set to destination if ReplaceMetadata is true
|
||||
// other value is UserMetadata is ignored and we preserve src.UserMetadata
|
||||
// NOTE: if you set this value to true and now metadata is present
|
||||
// in UserMetadata your destination object will not have any metadata
|
||||
// set.
|
||||
ReplaceMetadata bool
|
||||
|
||||
// `userTags` is the user defined object tags to be set on destination.
|
||||
// This will be set only if the `replaceTags` field is set to true.
|
||||
// Otherwise this field is ignored
|
||||
UserTags map[string]string
|
||||
ReplaceTags bool
|
||||
|
||||
// Specifies whether you want to apply a Legal Hold to the copied object.
|
||||
LegalHold LegalHoldStatus
|
||||
|
||||
// Object Retention related fields
|
||||
Mode RetentionMode
|
||||
RetainUntilDate time.Time
|
||||
|
||||
Size int64 // Needs to be specified if progress bar is specified.
|
||||
// Progress of the entire copy operation will be sent here.
|
||||
Progress io.Reader
|
||||
}
|
||||
|
||||
// Process custom-metadata to remove a `x-amz-meta-` prefix if
|
||||
// present and validate that keys are distinct (after this
|
||||
// prefix removal).
|
||||
func filterCustomMeta(userMeta map[string]string) map[string]string {
|
||||
m := make(map[string]string)
|
||||
for k, v := range userMeta {
|
||||
if strings.HasPrefix(strings.ToLower(k), "x-amz-meta-") {
|
||||
k = k[len("x-amz-meta-"):]
|
||||
}
|
||||
if _, ok := m[k]; ok {
|
||||
continue
|
||||
}
|
||||
m[k] = v
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// Marshal converts all the CopyDestOptions into their
|
||||
// equivalent HTTP header representation
|
||||
func (opts CopyDestOptions) Marshal(header http.Header) {
|
||||
const replaceDirective = "REPLACE"
|
||||
if opts.ReplaceTags {
|
||||
header.Set(amzTaggingHeaderDirective, replaceDirective)
|
||||
if tags := s3utils.TagEncode(opts.UserTags); tags != "" {
|
||||
header.Set(amzTaggingHeader, tags)
|
||||
}
|
||||
}
|
||||
|
||||
if opts.LegalHold != LegalHoldStatus("") {
|
||||
header.Set(amzLegalHoldHeader, opts.LegalHold.String())
|
||||
}
|
||||
|
||||
if opts.Mode != RetentionMode("") && !opts.RetainUntilDate.IsZero() {
|
||||
header.Set(amzLockMode, opts.Mode.String())
|
||||
header.Set(amzLockRetainUntil, opts.RetainUntilDate.Format(time.RFC3339))
|
||||
}
|
||||
|
||||
if opts.Encryption != nil {
|
||||
opts.Encryption.Marshal(header)
|
||||
}
|
||||
|
||||
if opts.ReplaceMetadata {
|
||||
header.Set("x-amz-metadata-directive", replaceDirective)
|
||||
for k, v := range filterCustomMeta(opts.UserMetadata) {
|
||||
if isAmzHeader(k) || isStandardHeader(k) || isStorageClassHeader(k) || isMinioHeader(k) {
|
||||
header.Set(k, v)
|
||||
} else {
|
||||
header.Set("x-amz-meta-"+k, v)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// toDestinationInfo returns a validated copyOptions object.
|
||||
func (opts CopyDestOptions) validate() (err error) {
|
||||
// Input validation.
|
||||
if err = s3utils.CheckValidBucketName(opts.Bucket); err != nil {
|
||||
return err
|
||||
}
|
||||
if err = s3utils.CheckValidObjectName(opts.Object); err != nil {
|
||||
return err
|
||||
}
|
||||
if opts.Progress != nil && opts.Size < 0 {
|
||||
return errInvalidArgument("For progress bar effective size needs to be specified")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// CopySrcOptions represents a source object to be copied, using
|
||||
// server-side copying APIs.
|
||||
type CopySrcOptions struct {
|
||||
Bucket, Object string
|
||||
VersionID string
|
||||
MatchETag string
|
||||
NoMatchETag string
|
||||
MatchModifiedSince time.Time
|
||||
MatchUnmodifiedSince time.Time
|
||||
MatchRange bool
|
||||
Start, End int64
|
||||
Encryption encrypt.ServerSide
|
||||
}
|
||||
|
||||
// Marshal converts all the CopySrcOptions into their
|
||||
// equivalent HTTP header representation
|
||||
func (opts CopySrcOptions) Marshal(header http.Header) {
|
||||
// Set the source header
|
||||
header.Set("x-amz-copy-source", s3utils.EncodePath(opts.Bucket+"/"+opts.Object))
|
||||
if opts.VersionID != "" {
|
||||
header.Set("x-amz-copy-source", s3utils.EncodePath(opts.Bucket+"/"+opts.Object)+"?versionId="+opts.VersionID)
|
||||
}
|
||||
|
||||
if opts.MatchETag != "" {
|
||||
header.Set("x-amz-copy-source-if-match", opts.MatchETag)
|
||||
}
|
||||
if opts.NoMatchETag != "" {
|
||||
header.Set("x-amz-copy-source-if-none-match", opts.NoMatchETag)
|
||||
}
|
||||
|
||||
if !opts.MatchModifiedSince.IsZero() {
|
||||
header.Set("x-amz-copy-source-if-modified-since", opts.MatchModifiedSince.Format(http.TimeFormat))
|
||||
}
|
||||
if !opts.MatchUnmodifiedSince.IsZero() {
|
||||
header.Set("x-amz-copy-source-if-unmodified-since", opts.MatchUnmodifiedSince.Format(http.TimeFormat))
|
||||
}
|
||||
|
||||
if opts.Encryption != nil {
|
||||
encrypt.SSECopy(opts.Encryption).Marshal(header)
|
||||
}
|
||||
}
|
||||
|
||||
func (opts CopySrcOptions) validate() (err error) {
|
||||
// Input validation.
|
||||
if err = s3utils.CheckValidBucketName(opts.Bucket); err != nil {
|
||||
return err
|
||||
}
|
||||
if err = s3utils.CheckValidObjectName(opts.Object); err != nil {
|
||||
return err
|
||||
}
|
||||
if opts.Start > opts.End || opts.Start < 0 {
|
||||
return errInvalidArgument("start must be non-negative, and start must be at most end.")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Low level implementation of CopyObject API, supports only upto 5GiB worth of copy.
|
||||
func (c *Client) copyObjectDo(ctx context.Context, srcBucket, srcObject, destBucket, destObject string,
|
||||
metadata map[string]string, srcOpts CopySrcOptions, dstOpts PutObjectOptions,
|
||||
) (ObjectInfo, error) {
|
||||
// Build headers.
|
||||
headers := make(http.Header)
|
||||
|
||||
// Set all the metadata headers.
|
||||
for k, v := range metadata {
|
||||
headers.Set(k, v)
|
||||
}
|
||||
if !dstOpts.Internal.ReplicationStatus.Empty() {
|
||||
headers.Set(amzBucketReplicationStatus, string(dstOpts.Internal.ReplicationStatus))
|
||||
}
|
||||
if !dstOpts.Internal.SourceMTime.IsZero() {
|
||||
headers.Set(minIOBucketSourceMTime, dstOpts.Internal.SourceMTime.Format(time.RFC3339Nano))
|
||||
}
|
||||
if dstOpts.Internal.SourceETag != "" {
|
||||
headers.Set(minIOBucketSourceETag, dstOpts.Internal.SourceETag)
|
||||
}
|
||||
if dstOpts.Internal.ReplicationRequest {
|
||||
headers.Set(minIOBucketReplicationRequest, "true")
|
||||
}
|
||||
if dstOpts.Internal.ReplicationValidityCheck {
|
||||
headers.Set(minIOBucketReplicationCheck, "true")
|
||||
}
|
||||
if !dstOpts.Internal.LegalholdTimestamp.IsZero() {
|
||||
headers.Set(minIOBucketReplicationObjectLegalHoldTimestamp, dstOpts.Internal.LegalholdTimestamp.Format(time.RFC3339Nano))
|
||||
}
|
||||
if !dstOpts.Internal.RetentionTimestamp.IsZero() {
|
||||
headers.Set(minIOBucketReplicationObjectRetentionTimestamp, dstOpts.Internal.RetentionTimestamp.Format(time.RFC3339Nano))
|
||||
}
|
||||
if !dstOpts.Internal.TaggingTimestamp.IsZero() {
|
||||
headers.Set(minIOBucketReplicationTaggingTimestamp, dstOpts.Internal.TaggingTimestamp.Format(time.RFC3339Nano))
|
||||
}
|
||||
|
||||
if len(dstOpts.UserTags) != 0 {
|
||||
headers.Set(amzTaggingHeader, s3utils.TagEncode(dstOpts.UserTags))
|
||||
}
|
||||
|
||||
reqMetadata := requestMetadata{
|
||||
bucketName: destBucket,
|
||||
objectName: destObject,
|
||||
customHeader: headers,
|
||||
}
|
||||
if dstOpts.Internal.SourceVersionID != "" {
|
||||
if dstOpts.Internal.SourceVersionID != nullVersionID {
|
||||
if _, err := uuid.Parse(dstOpts.Internal.SourceVersionID); err != nil {
|
||||
return ObjectInfo{}, errInvalidArgument(err.Error())
|
||||
}
|
||||
}
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("versionId", dstOpts.Internal.SourceVersionID)
|
||||
reqMetadata.queryValues = urlValues
|
||||
}
|
||||
|
||||
// Set the source header
|
||||
headers.Set("x-amz-copy-source", s3utils.EncodePath(srcBucket+"/"+srcObject))
|
||||
if srcOpts.VersionID != "" {
|
||||
headers.Set("x-amz-copy-source", s3utils.EncodePath(srcBucket+"/"+srcObject)+"?versionId="+srcOpts.VersionID)
|
||||
}
|
||||
// Send upload-part-copy request
|
||||
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return ObjectInfo{}, err
|
||||
}
|
||||
|
||||
// Check if we got an error response.
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return ObjectInfo{}, httpRespToErrorResponse(resp, srcBucket, srcObject)
|
||||
}
|
||||
|
||||
cpObjRes := copyObjectResult{}
|
||||
err = xmlDecoder(resp.Body, &cpObjRes)
|
||||
if err != nil {
|
||||
return ObjectInfo{}, err
|
||||
}
|
||||
|
||||
objInfo := ObjectInfo{
|
||||
Key: destObject,
|
||||
ETag: strings.Trim(cpObjRes.ETag, "\""),
|
||||
LastModified: cpObjRes.LastModified,
|
||||
}
|
||||
return objInfo, nil
|
||||
}
|
||||
|
||||
func (c *Client) copyObjectPartDo(ctx context.Context, srcBucket, srcObject, destBucket, destObject, uploadID string,
|
||||
partID int, startOffset, length int64, metadata map[string]string,
|
||||
) (p CompletePart, err error) {
|
||||
headers := make(http.Header)
|
||||
|
||||
// Set source
|
||||
headers.Set("x-amz-copy-source", s3utils.EncodePath(srcBucket+"/"+srcObject))
|
||||
|
||||
if startOffset < 0 {
|
||||
return p, errInvalidArgument("startOffset must be non-negative")
|
||||
}
|
||||
|
||||
if length >= 0 {
|
||||
headers.Set("x-amz-copy-source-range", fmt.Sprintf("bytes=%d-%d", startOffset, startOffset+length-1))
|
||||
}
|
||||
|
||||
for k, v := range metadata {
|
||||
headers.Set(k, v)
|
||||
}
|
||||
|
||||
queryValues := make(url.Values)
|
||||
queryValues.Set("partNumber", strconv.Itoa(partID))
|
||||
queryValues.Set("uploadId", uploadID)
|
||||
|
||||
resp, err := c.executeMethod(ctx, http.MethodPut, requestMetadata{
|
||||
bucketName: destBucket,
|
||||
objectName: destObject,
|
||||
customHeader: headers,
|
||||
queryValues: queryValues,
|
||||
})
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
|
||||
// Check if we got an error response.
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return p, httpRespToErrorResponse(resp, destBucket, destObject)
|
||||
}
|
||||
|
||||
// Decode copy-part response on success.
|
||||
cpObjRes := copyObjectResult{}
|
||||
err = xmlDecoder(resp.Body, &cpObjRes)
|
||||
if err != nil {
|
||||
return p, err
|
||||
}
|
||||
p.PartNumber, p.ETag = partID, cpObjRes.ETag
|
||||
return p, nil
|
||||
}
|
||||
|
||||
// uploadPartCopy - helper function to create a part in a multipart
|
||||
// upload via an upload-part-copy request
|
||||
// https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPartCopy.html
|
||||
func (c *Client) uploadPartCopy(ctx context.Context, bucket, object, uploadID string, partNumber int,
|
||||
headers http.Header,
|
||||
) (p CompletePart, err error) {
|
||||
// Build query parameters
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("partNumber", strconv.Itoa(partNumber))
|
||||
urlValues.Set("uploadId", uploadID)
|
||||
|
||||
// Send upload-part-copy request
|
||||
resp, err := c.executeMethod(ctx, http.MethodPut, requestMetadata{
|
||||
bucketName: bucket,
|
||||
objectName: object,
|
||||
customHeader: headers,
|
||||
queryValues: urlValues,
|
||||
})
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return p, err
|
||||
}
|
||||
|
||||
// Check if we got an error response.
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return p, httpRespToErrorResponse(resp, bucket, object)
|
||||
}
|
||||
|
||||
// Decode copy-part response on success.
|
||||
cpObjRes := copyObjectResult{}
|
||||
err = xmlDecoder(resp.Body, &cpObjRes)
|
||||
if err != nil {
|
||||
return p, err
|
||||
}
|
||||
p.PartNumber, p.ETag = partNumber, cpObjRes.ETag
|
||||
return p, nil
|
||||
}
|
||||
|
||||
// ComposeObject - creates an object using server-side copying
|
||||
// of existing objects. It takes a list of source objects (with optional offsets)
|
||||
// and concatenates them into a new object using only server-side copying
|
||||
// operations. Optionally takes progress reader hook for applications to
|
||||
// look at current progress.
|
||||
func (c *Client) ComposeObject(ctx context.Context, dst CopyDestOptions, srcs ...CopySrcOptions) (UploadInfo, error) {
|
||||
if len(srcs) < 1 || len(srcs) > maxPartsCount {
|
||||
return UploadInfo{}, errInvalidArgument("There must be as least one and up to 10000 source objects.")
|
||||
}
|
||||
|
||||
for _, src := range srcs {
|
||||
if err := src.validate(); err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
}
|
||||
|
||||
if err := dst.validate(); err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
|
||||
srcObjectInfos := make([]ObjectInfo, len(srcs))
|
||||
srcObjectSizes := make([]int64, len(srcs))
|
||||
var totalSize, totalParts int64
|
||||
var err error
|
||||
for i, src := range srcs {
|
||||
opts := StatObjectOptions{ServerSideEncryption: encrypt.SSE(src.Encryption), VersionID: src.VersionID}
|
||||
srcObjectInfos[i], err = c.StatObject(context.Background(), src.Bucket, src.Object, opts)
|
||||
if err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
|
||||
srcCopySize := srcObjectInfos[i].Size
|
||||
// Check if a segment is specified, and if so, is the
|
||||
// segment within object bounds?
|
||||
if src.MatchRange {
|
||||
// Since range is specified,
|
||||
// 0 <= src.start <= src.end
|
||||
// so only invalid case to check is:
|
||||
if src.End >= srcCopySize || src.Start < 0 {
|
||||
return UploadInfo{}, errInvalidArgument(
|
||||
fmt.Sprintf("CopySrcOptions %d has invalid segment-to-copy [%d, %d] (size is %d)",
|
||||
i, src.Start, src.End, srcCopySize))
|
||||
}
|
||||
srcCopySize = src.End - src.Start + 1
|
||||
}
|
||||
|
||||
// Only the last source may be less than `absMinPartSize`
|
||||
if srcCopySize < absMinPartSize && i < len(srcs)-1 {
|
||||
return UploadInfo{}, errInvalidArgument(
|
||||
fmt.Sprintf("CopySrcOptions %d is too small (%d) and it is not the last part", i, srcCopySize))
|
||||
}
|
||||
|
||||
// Is data to copy too large?
|
||||
totalSize += srcCopySize
|
||||
if totalSize > maxMultipartPutObjectSize {
|
||||
return UploadInfo{}, errInvalidArgument(fmt.Sprintf("Cannot compose an object of size %d (> 5TiB)", totalSize))
|
||||
}
|
||||
|
||||
// record source size
|
||||
srcObjectSizes[i] = srcCopySize
|
||||
|
||||
// calculate parts needed for current source
|
||||
totalParts += partsRequired(srcCopySize)
|
||||
// Do we need more parts than we are allowed?
|
||||
if totalParts > maxPartsCount {
|
||||
return UploadInfo{}, errInvalidArgument(fmt.Sprintf(
|
||||
"Your proposed compose object requires more than %d parts", maxPartsCount))
|
||||
}
|
||||
}
|
||||
|
||||
// Single source object case (i.e. when only one source is
|
||||
// involved, it is being copied wholly and at most 5GiB in
|
||||
// size, emptyfiles are also supported).
|
||||
if (totalParts == 1 && srcs[0].Start == -1 && totalSize <= maxPartSize) || (totalSize == 0) {
|
||||
return c.CopyObject(ctx, dst, srcs[0])
|
||||
}
|
||||
|
||||
// Now, handle multipart-copy cases.
|
||||
|
||||
// 1. Ensure that the object has not been changed while
|
||||
// we are copying data.
|
||||
for i, src := range srcs {
|
||||
src.MatchETag = srcObjectInfos[i].ETag
|
||||
}
|
||||
|
||||
// 2. Initiate a new multipart upload.
|
||||
|
||||
// Set user-metadata on the destination object. If no
|
||||
// user-metadata is specified, and there is only one source,
|
||||
// (only) then metadata from source is copied.
|
||||
var userMeta map[string]string
|
||||
if dst.ReplaceMetadata {
|
||||
userMeta = dst.UserMetadata
|
||||
} else {
|
||||
userMeta = srcObjectInfos[0].UserMetadata
|
||||
}
|
||||
|
||||
var userTags map[string]string
|
||||
if dst.ReplaceTags {
|
||||
userTags = dst.UserTags
|
||||
} else {
|
||||
userTags = srcObjectInfos[0].UserTags
|
||||
}
|
||||
|
||||
uploadID, err := c.newUploadID(ctx, dst.Bucket, dst.Object, PutObjectOptions{
|
||||
ServerSideEncryption: dst.Encryption,
|
||||
UserMetadata: userMeta,
|
||||
UserTags: userTags,
|
||||
Mode: dst.Mode,
|
||||
RetainUntilDate: dst.RetainUntilDate,
|
||||
LegalHold: dst.LegalHold,
|
||||
})
|
||||
if err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
|
||||
// 3. Perform copy part uploads
|
||||
objParts := []CompletePart{}
|
||||
partIndex := 1
|
||||
for i, src := range srcs {
|
||||
h := make(http.Header)
|
||||
src.Marshal(h)
|
||||
if dst.Encryption != nil && dst.Encryption.Type() == encrypt.SSEC {
|
||||
dst.Encryption.Marshal(h)
|
||||
}
|
||||
|
||||
// calculate start/end indices of parts after
|
||||
// splitting.
|
||||
startIdx, endIdx := calculateEvenSplits(srcObjectSizes[i], src)
|
||||
for j, start := range startIdx {
|
||||
end := endIdx[j]
|
||||
|
||||
// Add (or reset) source range header for
|
||||
// upload part copy request.
|
||||
h.Set("x-amz-copy-source-range",
|
||||
fmt.Sprintf("bytes=%d-%d", start, end))
|
||||
|
||||
// make upload-part-copy request
|
||||
complPart, err := c.uploadPartCopy(ctx, dst.Bucket,
|
||||
dst.Object, uploadID, partIndex, h)
|
||||
if err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
if dst.Progress != nil {
|
||||
io.CopyN(io.Discard, dst.Progress, end-start+1)
|
||||
}
|
||||
objParts = append(objParts, complPart)
|
||||
partIndex++
|
||||
}
|
||||
}
|
||||
|
||||
// 4. Make final complete-multipart request.
|
||||
uploadInfo, err := c.completeMultipartUpload(ctx, dst.Bucket, dst.Object, uploadID,
|
||||
completeMultipartUpload{Parts: objParts}, PutObjectOptions{ServerSideEncryption: dst.Encryption})
|
||||
if err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
|
||||
uploadInfo.Size = totalSize
|
||||
return uploadInfo, nil
|
||||
}
|
||||
|
||||
// partsRequired is maximum parts possible with
|
||||
// max part size of ceiling(maxMultipartPutObjectSize / (maxPartsCount - 1))
|
||||
func partsRequired(size int64) int64 {
|
||||
maxPartSize := maxMultipartPutObjectSize / (maxPartsCount - 1)
|
||||
r := size / int64(maxPartSize)
|
||||
if size%int64(maxPartSize) > 0 {
|
||||
r++
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
// calculateEvenSplits - computes splits for a source and returns
|
||||
// start and end index slices. Splits happen evenly to be sure that no
|
||||
// part is less than 5MiB, as that could fail the multipart request if
|
||||
// it is not the last part.
|
||||
func calculateEvenSplits(size int64, src CopySrcOptions) (startIndex, endIndex []int64) {
|
||||
if size == 0 {
|
||||
return
|
||||
}
|
||||
|
||||
reqParts := partsRequired(size)
|
||||
startIndex = make([]int64, reqParts)
|
||||
endIndex = make([]int64, reqParts)
|
||||
// Compute number of required parts `k`, as:
|
||||
//
|
||||
// k = ceiling(size / copyPartSize)
|
||||
//
|
||||
// Now, distribute the `size` bytes in the source into
|
||||
// k parts as evenly as possible:
|
||||
//
|
||||
// r parts sized (q+1) bytes, and
|
||||
// (k - r) parts sized q bytes, where
|
||||
//
|
||||
// size = q * k + r (by simple division of size by k,
|
||||
// so that 0 <= r < k)
|
||||
//
|
||||
start := src.Start
|
||||
if start == -1 {
|
||||
start = 0
|
||||
}
|
||||
quot, rem := size/reqParts, size%reqParts
|
||||
nextStart := start
|
||||
for j := int64(0); j < reqParts; j++ {
|
||||
curPartSize := quot
|
||||
if j < rem {
|
||||
curPartSize++
|
||||
}
|
||||
|
||||
cStart := nextStart
|
||||
cEnd := cStart + curPartSize - 1
|
||||
nextStart = cEnd + 1
|
||||
|
||||
startIndex[j], endIndex[j] = cStart, cEnd
|
||||
}
|
||||
return
|
||||
}
|
||||
+76
@@ -0,0 +1,76 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2017, 2018 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"net/http"
|
||||
)
|
||||
|
||||
// CopyObject - copy a source object into a new object
|
||||
func (c *Client) CopyObject(ctx context.Context, dst CopyDestOptions, src CopySrcOptions) (UploadInfo, error) {
|
||||
if err := src.validate(); err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
|
||||
if err := dst.validate(); err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
|
||||
header := make(http.Header)
|
||||
dst.Marshal(header)
|
||||
src.Marshal(header)
|
||||
|
||||
resp, err := c.executeMethod(ctx, http.MethodPut, requestMetadata{
|
||||
bucketName: dst.Bucket,
|
||||
objectName: dst.Object,
|
||||
customHeader: header,
|
||||
})
|
||||
if err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
defer closeResponse(resp)
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return UploadInfo{}, httpRespToErrorResponse(resp, dst.Bucket, dst.Object)
|
||||
}
|
||||
|
||||
// Update the progress properly after successful copy.
|
||||
if dst.Progress != nil {
|
||||
io.Copy(io.Discard, io.LimitReader(dst.Progress, dst.Size))
|
||||
}
|
||||
|
||||
cpObjRes := copyObjectResult{}
|
||||
if err = xmlDecoder(resp.Body, &cpObjRes); err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
|
||||
// extract lifecycle expiry date and rule ID
|
||||
expTime, ruleID := amzExpirationToExpiryDateRuleID(resp.Header.Get(amzExpiration))
|
||||
|
||||
return UploadInfo{
|
||||
Bucket: dst.Bucket,
|
||||
Key: dst.Object,
|
||||
LastModified: cpObjRes.LastModified,
|
||||
ETag: trimEtag(resp.Header.Get("ETag")),
|
||||
VersionID: resp.Header.Get(amzVersionID),
|
||||
Expiration: expTime,
|
||||
ExpirationRuleID: ruleID,
|
||||
}, nil
|
||||
}
|
||||
+256
@@ -0,0 +1,256 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2015-2020 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"encoding/xml"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// BucketInfo container for bucket metadata.
|
||||
type BucketInfo struct {
|
||||
// The name of the bucket.
|
||||
Name string `json:"name"`
|
||||
// Date the bucket was created.
|
||||
CreationDate time.Time `json:"creationDate"`
|
||||
}
|
||||
|
||||
// StringMap represents map with custom UnmarshalXML
|
||||
type StringMap map[string]string
|
||||
|
||||
// UnmarshalXML unmarshals the XML into a map of string to strings,
|
||||
// creating a key in the map for each tag and setting it's value to the
|
||||
// tags contents.
|
||||
//
|
||||
// The fact this function is on the pointer of Map is important, so that
|
||||
// if m is nil it can be initialized, which is often the case if m is
|
||||
// nested in another xml structural. This is also why the first thing done
|
||||
// on the first line is initialize it.
|
||||
func (m *StringMap) UnmarshalXML(d *xml.Decoder, _ xml.StartElement) error {
|
||||
*m = StringMap{}
|
||||
for {
|
||||
// Format is <key>value</key>
|
||||
var e struct {
|
||||
XMLName xml.Name
|
||||
Value string `xml:",chardata"`
|
||||
}
|
||||
err := d.Decode(&e)
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
(*m)[e.XMLName.Local] = e.Value
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// URLMap represents map with custom UnmarshalXML
|
||||
type URLMap map[string]string
|
||||
|
||||
// UnmarshalXML unmarshals the XML into a map of string to strings,
|
||||
// creating a key in the map for each tag and setting it's value to the
|
||||
// tags contents.
|
||||
//
|
||||
// The fact this function is on the pointer of Map is important, so that
|
||||
// if m is nil it can be initialized, which is often the case if m is
|
||||
// nested in another xml structural. This is also why the first thing done
|
||||
// on the first line is initialize it.
|
||||
func (m *URLMap) UnmarshalXML(d *xml.Decoder, se xml.StartElement) error {
|
||||
*m = URLMap{}
|
||||
var tgs string
|
||||
if err := d.DecodeElement(&tgs, &se); err != nil {
|
||||
if err == io.EOF {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
for tgs != "" {
|
||||
var key string
|
||||
key, tgs, _ = stringsCut(tgs, "&")
|
||||
if key == "" {
|
||||
continue
|
||||
}
|
||||
key, value, _ := stringsCut(key, "=")
|
||||
key, err := url.QueryUnescape(key)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
value, err = url.QueryUnescape(value)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
(*m)[key] = value
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// stringsCut slices s around the first instance of sep,
|
||||
// returning the text before and after sep.
|
||||
// The found result reports whether sep appears in s.
|
||||
// If sep does not appear in s, cut returns s, "", false.
|
||||
func stringsCut(s, sep string) (before, after string, found bool) {
|
||||
if i := strings.Index(s, sep); i >= 0 {
|
||||
return s[:i], s[i+len(sep):], true
|
||||
}
|
||||
return s, "", false
|
||||
}
|
||||
|
||||
// Owner name.
|
||||
type Owner struct {
|
||||
XMLName xml.Name `xml:"Owner" json:"owner"`
|
||||
DisplayName string `xml:"ID" json:"name"`
|
||||
ID string `xml:"DisplayName" json:"id"`
|
||||
}
|
||||
|
||||
// UploadInfo contains information about the
|
||||
// newly uploaded or copied object.
|
||||
type UploadInfo struct {
|
||||
Bucket string
|
||||
Key string
|
||||
ETag string
|
||||
Size int64
|
||||
LastModified time.Time
|
||||
Location string
|
||||
VersionID string
|
||||
|
||||
// Lifecycle expiry-date and ruleID associated with the expiry
|
||||
// not to be confused with `Expires` HTTP header.
|
||||
Expiration time.Time
|
||||
ExpirationRuleID string
|
||||
|
||||
// Verified checksum values, if any.
|
||||
// Values are base64 (standard) encoded.
|
||||
// For multipart objects this is a checksum of the checksum of each part.
|
||||
ChecksumCRC32 string
|
||||
ChecksumCRC32C string
|
||||
ChecksumSHA1 string
|
||||
ChecksumSHA256 string
|
||||
ChecksumCRC64NVME string
|
||||
}
|
||||
|
||||
// RestoreInfo contains information of the restore operation of an archived object
|
||||
type RestoreInfo struct {
|
||||
// Is the restoring operation is still ongoing
|
||||
OngoingRestore bool
|
||||
// When the restored copy of the archived object will be removed
|
||||
ExpiryTime time.Time
|
||||
}
|
||||
|
||||
// ObjectInfo container for object metadata.
|
||||
type ObjectInfo struct {
|
||||
// An ETag is optionally set to md5sum of an object. In case of multipart objects,
|
||||
// ETag is of the form MD5SUM-N where MD5SUM is md5sum of all individual md5sums of
|
||||
// each parts concatenated into one string.
|
||||
ETag string `json:"etag"`
|
||||
|
||||
Key string `json:"name"` // Name of the object
|
||||
LastModified time.Time `json:"lastModified"` // Date and time the object was last modified.
|
||||
Size int64 `json:"size"` // Size in bytes of the object.
|
||||
ContentType string `json:"contentType"` // A standard MIME type describing the format of the object data.
|
||||
Expires time.Time `json:"expires"` // The date and time at which the object is no longer able to be cached.
|
||||
|
||||
// Collection of additional metadata on the object.
|
||||
// eg: x-amz-meta-*, content-encoding etc.
|
||||
Metadata http.Header `json:"metadata" xml:"-"`
|
||||
|
||||
// x-amz-meta-* headers stripped "x-amz-meta-" prefix containing the first value.
|
||||
// Only returned by MinIO servers.
|
||||
UserMetadata StringMap `json:"userMetadata,omitempty"`
|
||||
|
||||
// x-amz-tagging values in their k/v values.
|
||||
// Only returned by MinIO servers.
|
||||
UserTags URLMap `json:"userTags,omitempty" xml:"UserTags"`
|
||||
|
||||
// x-amz-tagging-count value
|
||||
UserTagCount int
|
||||
|
||||
// Owner name.
|
||||
Owner Owner
|
||||
|
||||
// ACL grant.
|
||||
Grant []Grant
|
||||
|
||||
// The class of storage used to store the object.
|
||||
StorageClass string `json:"storageClass"`
|
||||
|
||||
// Versioning related information
|
||||
IsLatest bool
|
||||
IsDeleteMarker bool
|
||||
VersionID string `xml:"VersionId"`
|
||||
|
||||
// x-amz-replication-status value is either in one of the following states
|
||||
// - COMPLETED
|
||||
// - PENDING
|
||||
// - FAILED
|
||||
// - REPLICA (on the destination)
|
||||
ReplicationStatus string `xml:"ReplicationStatus"`
|
||||
// set to true if delete marker has backing object version on target, and eligible to replicate
|
||||
ReplicationReady bool
|
||||
// Lifecycle expiry-date and ruleID associated with the expiry
|
||||
// not to be confused with `Expires` HTTP header.
|
||||
Expiration time.Time
|
||||
ExpirationRuleID string
|
||||
|
||||
Restore *RestoreInfo
|
||||
|
||||
// Checksum values
|
||||
ChecksumCRC32 string
|
||||
ChecksumCRC32C string
|
||||
ChecksumSHA1 string
|
||||
ChecksumSHA256 string
|
||||
ChecksumCRC64NVME string
|
||||
|
||||
Internal *struct {
|
||||
K int // Data blocks
|
||||
M int // Parity blocks
|
||||
} `xml:"Internal"`
|
||||
|
||||
// Error
|
||||
Err error `json:"-"`
|
||||
}
|
||||
|
||||
// ObjectMultipartInfo container for multipart object metadata.
|
||||
type ObjectMultipartInfo struct {
|
||||
// Date and time at which the multipart upload was initiated.
|
||||
Initiated time.Time `type:"timestamp" timestampFormat:"iso8601"`
|
||||
|
||||
Initiator initiator
|
||||
Owner owner
|
||||
|
||||
// The type of storage to use for the object. Defaults to 'STANDARD'.
|
||||
StorageClass string
|
||||
|
||||
// Key of the object for which the multipart upload was initiated.
|
||||
Key string
|
||||
|
||||
// Size in bytes of the object.
|
||||
Size int64
|
||||
|
||||
// Upload ID that identifies the multipart upload.
|
||||
UploadID string `xml:"UploadId"`
|
||||
|
||||
// Error
|
||||
Err error
|
||||
}
|
||||
+284
@@ -0,0 +1,284 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2015-2020 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/xml"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
)
|
||||
|
||||
/* **** SAMPLE ERROR RESPONSE ****
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<Error>
|
||||
<Code>AccessDenied</Code>
|
||||
<Message>Access Denied</Message>
|
||||
<BucketName>bucketName</BucketName>
|
||||
<Key>objectName</Key>
|
||||
<RequestId>F19772218238A85A</RequestId>
|
||||
<HostId>GuWkjyviSiGHizehqpmsD1ndz5NClSP19DOT+s2mv7gXGQ8/X1lhbDGiIJEXpGFD</HostId>
|
||||
</Error>
|
||||
*/
|
||||
|
||||
// ErrorResponse - Is the typed error returned by all API operations.
|
||||
// ErrorResponse struct should be comparable since it is compared inside
|
||||
// golang http API (https://github.com/golang/go/issues/29768)
|
||||
type ErrorResponse struct {
|
||||
XMLName xml.Name `xml:"Error" json:"-"`
|
||||
Code string
|
||||
Message string
|
||||
BucketName string
|
||||
Key string
|
||||
Resource string
|
||||
RequestID string `xml:"RequestId"`
|
||||
HostID string `xml:"HostId"`
|
||||
|
||||
// Region where the bucket is located. This header is returned
|
||||
// only in HEAD bucket and ListObjects response.
|
||||
Region string
|
||||
|
||||
// Captures the server string returned in response header.
|
||||
Server string
|
||||
|
||||
// Underlying HTTP status code for the returned error
|
||||
StatusCode int `xml:"-" json:"-"`
|
||||
}
|
||||
|
||||
// ToErrorResponse - Returns parsed ErrorResponse struct from body and
|
||||
// http headers.
|
||||
//
|
||||
// For example:
|
||||
//
|
||||
// import s3 "github.com/minio/minio-go/v7"
|
||||
// ...
|
||||
// ...
|
||||
// reader, stat, err := s3.GetObject(...)
|
||||
// if err != nil {
|
||||
// resp := s3.ToErrorResponse(err)
|
||||
// }
|
||||
// ...
|
||||
func ToErrorResponse(err error) ErrorResponse {
|
||||
switch err := err.(type) {
|
||||
case ErrorResponse:
|
||||
return err
|
||||
default:
|
||||
return ErrorResponse{}
|
||||
}
|
||||
}
|
||||
|
||||
// Error - Returns S3 error string.
|
||||
func (e ErrorResponse) Error() string {
|
||||
if e.Message == "" {
|
||||
msg, ok := s3ErrorResponseMap[e.Code]
|
||||
if !ok {
|
||||
msg = fmt.Sprintf("Error response code %s.", e.Code)
|
||||
}
|
||||
return msg
|
||||
}
|
||||
return e.Message
|
||||
}
|
||||
|
||||
// Common string for errors to report issue location in unexpected
|
||||
// cases.
|
||||
const (
|
||||
reportIssue = "Please report this issue at https://github.com/minio/minio-go/issues."
|
||||
)
|
||||
|
||||
// xmlDecodeAndBody reads the whole body up to 1MB and
|
||||
// tries to XML decode it into v.
|
||||
// The body that was read and any error from reading or decoding is returned.
|
||||
func xmlDecodeAndBody(bodyReader io.Reader, v interface{}) ([]byte, error) {
|
||||
// read the whole body (up to 1MB)
|
||||
const maxBodyLength = 1 << 20
|
||||
body, err := io.ReadAll(io.LimitReader(bodyReader, maxBodyLength))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return bytes.TrimSpace(body), xmlDecoder(bytes.NewReader(body), v)
|
||||
}
|
||||
|
||||
// httpRespToErrorResponse returns a new encoded ErrorResponse
|
||||
// structure as error.
|
||||
func httpRespToErrorResponse(resp *http.Response, bucketName, objectName string) error {
|
||||
if resp == nil {
|
||||
msg := "Empty http response. " + reportIssue
|
||||
return errInvalidArgument(msg)
|
||||
}
|
||||
|
||||
errResp := ErrorResponse{
|
||||
StatusCode: resp.StatusCode,
|
||||
Server: resp.Header.Get("Server"),
|
||||
}
|
||||
|
||||
errBody, err := xmlDecodeAndBody(resp.Body, &errResp)
|
||||
// Xml decoding failed with no body, fall back to HTTP headers.
|
||||
if err != nil {
|
||||
switch resp.StatusCode {
|
||||
case http.StatusNotFound:
|
||||
if objectName == "" {
|
||||
errResp = ErrorResponse{
|
||||
StatusCode: resp.StatusCode,
|
||||
Code: "NoSuchBucket",
|
||||
Message: "The specified bucket does not exist.",
|
||||
BucketName: bucketName,
|
||||
}
|
||||
} else {
|
||||
errResp = ErrorResponse{
|
||||
StatusCode: resp.StatusCode,
|
||||
Code: "NoSuchKey",
|
||||
Message: "The specified key does not exist.",
|
||||
BucketName: bucketName,
|
||||
Key: objectName,
|
||||
}
|
||||
}
|
||||
case http.StatusForbidden:
|
||||
errResp = ErrorResponse{
|
||||
StatusCode: resp.StatusCode,
|
||||
Code: "AccessDenied",
|
||||
Message: "Access Denied.",
|
||||
BucketName: bucketName,
|
||||
Key: objectName,
|
||||
}
|
||||
case http.StatusConflict:
|
||||
errResp = ErrorResponse{
|
||||
StatusCode: resp.StatusCode,
|
||||
Code: "Conflict",
|
||||
Message: "Bucket not empty.",
|
||||
BucketName: bucketName,
|
||||
}
|
||||
case http.StatusPreconditionFailed:
|
||||
errResp = ErrorResponse{
|
||||
StatusCode: resp.StatusCode,
|
||||
Code: "PreconditionFailed",
|
||||
Message: s3ErrorResponseMap["PreconditionFailed"],
|
||||
BucketName: bucketName,
|
||||
Key: objectName,
|
||||
}
|
||||
default:
|
||||
msg := resp.Status
|
||||
if len(errBody) > 0 {
|
||||
msg = string(errBody)
|
||||
if len(msg) > 1024 {
|
||||
msg = msg[:1024] + "..."
|
||||
}
|
||||
}
|
||||
errResp = ErrorResponse{
|
||||
StatusCode: resp.StatusCode,
|
||||
Code: resp.Status,
|
||||
Message: msg,
|
||||
BucketName: bucketName,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
code := resp.Header.Get("x-minio-error-code")
|
||||
if code != "" {
|
||||
errResp.Code = code
|
||||
}
|
||||
desc := resp.Header.Get("x-minio-error-desc")
|
||||
if desc != "" {
|
||||
errResp.Message = strings.Trim(desc, `"`)
|
||||
}
|
||||
|
||||
// Save hostID, requestID and region information
|
||||
// from headers if not available through error XML.
|
||||
if errResp.RequestID == "" {
|
||||
errResp.RequestID = resp.Header.Get("x-amz-request-id")
|
||||
}
|
||||
if errResp.HostID == "" {
|
||||
errResp.HostID = resp.Header.Get("x-amz-id-2")
|
||||
}
|
||||
if errResp.Region == "" {
|
||||
errResp.Region = resp.Header.Get("x-amz-bucket-region")
|
||||
}
|
||||
if errResp.Code == "InvalidRegion" && errResp.Region != "" {
|
||||
errResp.Message = fmt.Sprintf("Region does not match, expecting region ‘%s’.", errResp.Region)
|
||||
}
|
||||
|
||||
return errResp
|
||||
}
|
||||
|
||||
// errTransferAccelerationBucket - bucket name is invalid to be used with transfer acceleration.
|
||||
func errTransferAccelerationBucket(bucketName string) error {
|
||||
return ErrorResponse{
|
||||
StatusCode: http.StatusBadRequest,
|
||||
Code: "InvalidArgument",
|
||||
Message: "The name of the bucket used for Transfer Acceleration must be DNS-compliant and must not contain periods ‘.’.",
|
||||
BucketName: bucketName,
|
||||
}
|
||||
}
|
||||
|
||||
// errEntityTooLarge - Input size is larger than supported maximum.
|
||||
func errEntityTooLarge(totalSize, maxObjectSize int64, bucketName, objectName string) error {
|
||||
msg := fmt.Sprintf("Your proposed upload size ‘%d’ exceeds the maximum allowed object size ‘%d’ for single PUT operation.", totalSize, maxObjectSize)
|
||||
return ErrorResponse{
|
||||
StatusCode: http.StatusBadRequest,
|
||||
Code: "EntityTooLarge",
|
||||
Message: msg,
|
||||
BucketName: bucketName,
|
||||
Key: objectName,
|
||||
}
|
||||
}
|
||||
|
||||
// errEntityTooSmall - Input size is smaller than supported minimum.
|
||||
func errEntityTooSmall(totalSize int64, bucketName, objectName string) error {
|
||||
msg := fmt.Sprintf("Your proposed upload size ‘%d’ is below the minimum allowed object size ‘0B’ for single PUT operation.", totalSize)
|
||||
return ErrorResponse{
|
||||
StatusCode: http.StatusBadRequest,
|
||||
Code: "EntityTooSmall",
|
||||
Message: msg,
|
||||
BucketName: bucketName,
|
||||
Key: objectName,
|
||||
}
|
||||
}
|
||||
|
||||
// errUnexpectedEOF - Unexpected end of file reached.
|
||||
func errUnexpectedEOF(totalRead, totalSize int64, bucketName, objectName string) error {
|
||||
msg := fmt.Sprintf("Data read ‘%d’ is not equal to the size ‘%d’ of the input Reader.", totalRead, totalSize)
|
||||
return ErrorResponse{
|
||||
StatusCode: http.StatusBadRequest,
|
||||
Code: "UnexpectedEOF",
|
||||
Message: msg,
|
||||
BucketName: bucketName,
|
||||
Key: objectName,
|
||||
}
|
||||
}
|
||||
|
||||
// errInvalidArgument - Invalid argument response.
|
||||
func errInvalidArgument(message string) error {
|
||||
return ErrorResponse{
|
||||
StatusCode: http.StatusBadRequest,
|
||||
Code: "InvalidArgument",
|
||||
Message: message,
|
||||
RequestID: "minio",
|
||||
}
|
||||
}
|
||||
|
||||
// errAPINotSupported - API not supported response
|
||||
// The specified API call is not supported
|
||||
func errAPINotSupported(message string) error {
|
||||
return ErrorResponse{
|
||||
StatusCode: http.StatusNotImplemented,
|
||||
Code: "APINotSupported",
|
||||
Message: message,
|
||||
RequestID: "minio",
|
||||
}
|
||||
}
|
||||
+152
@@ -0,0 +1,152 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2018 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/xml"
|
||||
"net/http"
|
||||
"net/url"
|
||||
)
|
||||
|
||||
// Grantee represents the person being granted permissions.
|
||||
type Grantee struct {
|
||||
XMLName xml.Name `xml:"Grantee"`
|
||||
ID string `xml:"ID"`
|
||||
DisplayName string `xml:"DisplayName"`
|
||||
URI string `xml:"URI"`
|
||||
}
|
||||
|
||||
// Grant holds grant information
|
||||
type Grant struct {
|
||||
XMLName xml.Name `xml:"Grant"`
|
||||
Grantee Grantee
|
||||
Permission string `xml:"Permission"`
|
||||
}
|
||||
|
||||
// AccessControlList contains the set of grantees and the permissions assigned to each grantee.
|
||||
type AccessControlList struct {
|
||||
XMLName xml.Name `xml:"AccessControlList"`
|
||||
Grant []Grant
|
||||
Permission string `xml:"Permission"`
|
||||
}
|
||||
|
||||
type accessControlPolicy struct {
|
||||
XMLName xml.Name `xml:"AccessControlPolicy"`
|
||||
Owner Owner
|
||||
AccessControlList AccessControlList
|
||||
}
|
||||
|
||||
// GetObjectACL get object ACLs
|
||||
func (c *Client) GetObjectACL(ctx context.Context, bucketName, objectName string) (*ObjectInfo, error) {
|
||||
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
objectName: objectName,
|
||||
queryValues: url.Values{
|
||||
"acl": []string{""},
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer closeResponse(resp)
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return nil, httpRespToErrorResponse(resp, bucketName, objectName)
|
||||
}
|
||||
|
||||
res := &accessControlPolicy{}
|
||||
|
||||
if err := xmlDecoder(resp.Body, res); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
objInfo, err := c.StatObject(ctx, bucketName, objectName, StatObjectOptions{})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
objInfo.Owner.DisplayName = res.Owner.DisplayName
|
||||
objInfo.Owner.ID = res.Owner.ID
|
||||
|
||||
objInfo.Grant = append(objInfo.Grant, res.AccessControlList.Grant...)
|
||||
|
||||
cannedACL := getCannedACL(res)
|
||||
if cannedACL != "" {
|
||||
objInfo.Metadata.Add("X-Amz-Acl", cannedACL)
|
||||
return &objInfo, nil
|
||||
}
|
||||
|
||||
grantACL := getAmzGrantACL(res)
|
||||
for k, v := range grantACL {
|
||||
objInfo.Metadata[k] = v
|
||||
}
|
||||
|
||||
return &objInfo, nil
|
||||
}
|
||||
|
||||
func getCannedACL(aCPolicy *accessControlPolicy) string {
|
||||
grants := aCPolicy.AccessControlList.Grant
|
||||
|
||||
switch {
|
||||
case len(grants) == 1:
|
||||
if grants[0].Grantee.URI == "" && grants[0].Permission == "FULL_CONTROL" {
|
||||
return "private"
|
||||
}
|
||||
case len(grants) == 2:
|
||||
for _, g := range grants {
|
||||
if g.Grantee.URI == "http://acs.amazonaws.com/groups/global/AuthenticatedUsers" && g.Permission == "READ" {
|
||||
return "authenticated-read"
|
||||
}
|
||||
if g.Grantee.URI == "http://acs.amazonaws.com/groups/global/AllUsers" && g.Permission == "READ" {
|
||||
return "public-read"
|
||||
}
|
||||
if g.Permission == "READ" && g.Grantee.ID == aCPolicy.Owner.ID {
|
||||
return "bucket-owner-read"
|
||||
}
|
||||
}
|
||||
case len(grants) == 3:
|
||||
for _, g := range grants {
|
||||
if g.Grantee.URI == "http://acs.amazonaws.com/groups/global/AllUsers" && g.Permission == "WRITE" {
|
||||
return "public-read-write"
|
||||
}
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func getAmzGrantACL(aCPolicy *accessControlPolicy) map[string][]string {
|
||||
grants := aCPolicy.AccessControlList.Grant
|
||||
res := map[string][]string{}
|
||||
|
||||
for _, g := range grants {
|
||||
switch {
|
||||
case g.Permission == "READ":
|
||||
res["X-Amz-Grant-Read"] = append(res["X-Amz-Grant-Read"], "id="+g.Grantee.ID)
|
||||
case g.Permission == "WRITE":
|
||||
res["X-Amz-Grant-Write"] = append(res["X-Amz-Grant-Write"], "id="+g.Grantee.ID)
|
||||
case g.Permission == "READ_ACP":
|
||||
res["X-Amz-Grant-Read-Acp"] = append(res["X-Amz-Grant-Read-Acp"], "id="+g.Grantee.ID)
|
||||
case g.Permission == "WRITE_ACP":
|
||||
res["X-Amz-Grant-Write-Acp"] = append(res["X-Amz-Grant-Write-Acp"], "id="+g.Grantee.ID)
|
||||
case g.Permission == "FULL_CONTROL":
|
||||
res["X-Amz-Grant-Full-Control"] = append(res["X-Amz-Grant-Full-Control"], "id="+g.Grantee.ID)
|
||||
}
|
||||
}
|
||||
return res
|
||||
}
|
||||
+201
@@ -0,0 +1,201 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2020 MinIO, Inc.
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/xml"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"github.com/minio/minio-go/v7/pkg/encrypt"
|
||||
"github.com/minio/minio-go/v7/pkg/s3utils"
|
||||
)
|
||||
|
||||
// ObjectAttributesOptions are options used for the GetObjectAttributes API
|
||||
//
|
||||
// - MaxParts
|
||||
// How many parts the caller wants to be returned (default: 1000)
|
||||
//
|
||||
// - VersionID
|
||||
// The object version you want to attributes for
|
||||
//
|
||||
// - PartNumberMarker
|
||||
// the listing will start AFTER the part matching PartNumberMarker
|
||||
//
|
||||
// - ServerSideEncryption
|
||||
// The server-side encryption algorithm used when storing this object in Minio
|
||||
type ObjectAttributesOptions struct {
|
||||
MaxParts int
|
||||
VersionID string
|
||||
PartNumberMarker int
|
||||
ServerSideEncryption encrypt.ServerSide
|
||||
}
|
||||
|
||||
// ObjectAttributes is the response object returned by the GetObjectAttributes API
|
||||
//
|
||||
// - VersionID
|
||||
// The object version
|
||||
//
|
||||
// - LastModified
|
||||
// The last time the object was modified
|
||||
//
|
||||
// - ObjectAttributesResponse
|
||||
// Contains more information about the object
|
||||
type ObjectAttributes struct {
|
||||
VersionID string
|
||||
LastModified time.Time
|
||||
ObjectAttributesResponse
|
||||
}
|
||||
|
||||
// ObjectAttributesResponse contains details returned by the GetObjectAttributes API
|
||||
//
|
||||
// Noteworthy fields:
|
||||
//
|
||||
// - ObjectParts.PartsCount
|
||||
// Contains the total part count for the object (not the current response)
|
||||
//
|
||||
// - ObjectParts.PartNumberMarker
|
||||
// Pagination of parts will begin at (but not include) PartNumberMarker
|
||||
//
|
||||
// - ObjectParts.NextPartNumberMarket
|
||||
// The next PartNumberMarker to be used in order to continue pagination
|
||||
//
|
||||
// - ObjectParts.IsTruncated
|
||||
// Indicates if the last part is included in the request (does not check if parts are missing from the start of the list, ONLY the end)
|
||||
//
|
||||
// - ObjectParts.MaxParts
|
||||
// Reflects the MaxParts used by the caller or the default MaxParts value of the API
|
||||
type ObjectAttributesResponse struct {
|
||||
ETag string `xml:",omitempty"`
|
||||
StorageClass string
|
||||
ObjectSize int
|
||||
Checksum struct {
|
||||
ChecksumCRC32 string `xml:",omitempty"`
|
||||
ChecksumCRC32C string `xml:",omitempty"`
|
||||
ChecksumSHA1 string `xml:",omitempty"`
|
||||
ChecksumSHA256 string `xml:",omitempty"`
|
||||
}
|
||||
ObjectParts struct {
|
||||
PartsCount int
|
||||
PartNumberMarker int
|
||||
NextPartNumberMarker int
|
||||
MaxParts int
|
||||
IsTruncated bool
|
||||
Parts []*ObjectAttributePart `xml:"Part"`
|
||||
}
|
||||
}
|
||||
|
||||
// ObjectAttributePart is used by ObjectAttributesResponse to describe an object part
|
||||
type ObjectAttributePart struct {
|
||||
ChecksumCRC32 string `xml:",omitempty"`
|
||||
ChecksumCRC32C string `xml:",omitempty"`
|
||||
ChecksumSHA1 string `xml:",omitempty"`
|
||||
ChecksumSHA256 string `xml:",omitempty"`
|
||||
PartNumber int
|
||||
Size int
|
||||
}
|
||||
|
||||
func (o *ObjectAttributes) parseResponse(resp *http.Response) (err error) {
|
||||
mod, err := parseRFC7231Time(resp.Header.Get("Last-Modified"))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
o.LastModified = mod
|
||||
o.VersionID = resp.Header.Get(amzVersionID)
|
||||
|
||||
response := new(ObjectAttributesResponse)
|
||||
if err := xml.NewDecoder(resp.Body).Decode(response); err != nil {
|
||||
return err
|
||||
}
|
||||
o.ObjectAttributesResponse = *response
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
// GetObjectAttributes API combines HeadObject and ListParts.
|
||||
// More details on usage can be found in the documentation for ObjectAttributesOptions{}
|
||||
func (c *Client) GetObjectAttributes(ctx context.Context, bucketName, objectName string, opts ObjectAttributesOptions) (*ObjectAttributes, error) {
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if err := s3utils.CheckValidObjectName(objectName); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Add("attributes", "")
|
||||
if opts.VersionID != "" {
|
||||
urlValues.Add("versionId", opts.VersionID)
|
||||
}
|
||||
|
||||
headers := make(http.Header)
|
||||
headers.Set(amzObjectAttributes, GetObjectAttributesTags)
|
||||
|
||||
if opts.PartNumberMarker > 0 {
|
||||
headers.Set(amzPartNumberMarker, strconv.Itoa(opts.PartNumberMarker))
|
||||
}
|
||||
|
||||
if opts.MaxParts > 0 {
|
||||
headers.Set(amzMaxParts, strconv.Itoa(opts.MaxParts))
|
||||
} else {
|
||||
headers.Set(amzMaxParts, strconv.Itoa(GetObjectAttributesMaxParts))
|
||||
}
|
||||
|
||||
if opts.ServerSideEncryption != nil {
|
||||
opts.ServerSideEncryption.Marshal(headers)
|
||||
}
|
||||
|
||||
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
objectName: objectName,
|
||||
queryValues: urlValues,
|
||||
contentSHA256Hex: emptySHA256Hex,
|
||||
customHeader: headers,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
defer closeResponse(resp)
|
||||
|
||||
hasEtag := resp.Header.Get(ETag)
|
||||
if hasEtag != "" {
|
||||
return nil, errors.New("getObjectAttributes is not supported by the current endpoint version")
|
||||
}
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
ER := new(ErrorResponse)
|
||||
if err := xml.NewDecoder(resp.Body).Decode(ER); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return nil, *ER
|
||||
}
|
||||
|
||||
OA := new(ObjectAttributes)
|
||||
err = OA.parseResponse(resp)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return OA, nil
|
||||
}
|
||||
+127
@@ -0,0 +1,127 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2015-2017 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
|
||||
"github.com/minio/minio-go/v7/pkg/s3utils"
|
||||
)
|
||||
|
||||
// FGetObject - download contents of an object to a local file.
|
||||
// The options can be used to specify the GET request further.
|
||||
func (c *Client) FGetObject(ctx context.Context, bucketName, objectName, filePath string, opts GetObjectOptions) error {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := s3utils.CheckValidObjectName(objectName); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Verify if destination already exists.
|
||||
st, err := os.Stat(filePath)
|
||||
if err == nil {
|
||||
// If the destination exists and is a directory.
|
||||
if st.IsDir() {
|
||||
return errInvalidArgument("fileName is a directory.")
|
||||
}
|
||||
}
|
||||
|
||||
// Proceed if file does not exist. return for all other errors.
|
||||
if err != nil {
|
||||
if !os.IsNotExist(err) {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
// Extract top level directory.
|
||||
objectDir, _ := filepath.Split(filePath)
|
||||
if objectDir != "" {
|
||||
// Create any missing top level directories.
|
||||
if err := os.MkdirAll(objectDir, 0o700); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
// Gather md5sum.
|
||||
objectStat, err := c.StatObject(ctx, bucketName, objectName, StatObjectOptions(opts))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Write to a temporary file "fileName.part.minio" before saving.
|
||||
filePartPath := filePath + sum256Hex([]byte(objectStat.ETag)) + ".part.minio"
|
||||
|
||||
// If exists, open in append mode. If not create it as a part file.
|
||||
filePart, err := os.OpenFile(filePartPath, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0o600)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// If we return early with an error, be sure to close and delete
|
||||
// filePart. If we have an error along the way there is a chance
|
||||
// that filePart is somehow damaged, and we should discard it.
|
||||
closeAndRemove := true
|
||||
defer func() {
|
||||
if closeAndRemove {
|
||||
_ = filePart.Close()
|
||||
_ = os.Remove(filePartPath)
|
||||
}
|
||||
}()
|
||||
|
||||
// Issue Stat to get the current offset.
|
||||
st, err = filePart.Stat()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Initialize get object request headers to set the
|
||||
// appropriate range offsets to read from.
|
||||
if st.Size() > 0 {
|
||||
opts.SetRange(st.Size(), 0)
|
||||
}
|
||||
|
||||
// Seek to current position for incoming reader.
|
||||
objectReader, objectStat, _, err := c.getObject(ctx, bucketName, objectName, opts)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Write to the part file.
|
||||
if _, err = io.CopyN(filePart, objectReader, objectStat.Size); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Close the file before rename, this is specifically needed for Windows users.
|
||||
closeAndRemove = false
|
||||
if err = filePart.Close(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Safely completed. Now commit by renaming to actual filename.
|
||||
if err = os.Rename(filePartPath, filePath); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Return.
|
||||
return nil
|
||||
}
|
||||
+699
@@ -0,0 +1,699 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2015-2020 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"sync"
|
||||
|
||||
"github.com/minio/minio-go/v7/pkg/s3utils"
|
||||
)
|
||||
|
||||
// GetObject wrapper function that accepts a request context
|
||||
func (c *Client) GetObject(ctx context.Context, bucketName, objectName string, opts GetObjectOptions) (*Object, error) {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return nil, ErrorResponse{
|
||||
StatusCode: http.StatusBadRequest,
|
||||
Code: "InvalidBucketName",
|
||||
Message: err.Error(),
|
||||
}
|
||||
}
|
||||
if err := s3utils.CheckValidObjectName(objectName); err != nil {
|
||||
return nil, ErrorResponse{
|
||||
StatusCode: http.StatusBadRequest,
|
||||
Code: "XMinioInvalidObjectName",
|
||||
Message: err.Error(),
|
||||
}
|
||||
}
|
||||
|
||||
gctx, cancel := context.WithCancel(ctx)
|
||||
|
||||
// Detect if snowball is server location we are talking to.
|
||||
var snowball bool
|
||||
if location, ok := c.bucketLocCache.Get(bucketName); ok {
|
||||
snowball = location == "snowball"
|
||||
}
|
||||
|
||||
var (
|
||||
err error
|
||||
httpReader io.ReadCloser
|
||||
objectInfo ObjectInfo
|
||||
totalRead int
|
||||
)
|
||||
|
||||
// Create request channel.
|
||||
reqCh := make(chan getRequest)
|
||||
// Create response channel.
|
||||
resCh := make(chan getResponse)
|
||||
|
||||
// This routine feeds partial object data as and when the caller reads.
|
||||
go func() {
|
||||
defer close(resCh)
|
||||
defer func() {
|
||||
// Close the http response body before returning.
|
||||
// This ends the connection with the server.
|
||||
if httpReader != nil {
|
||||
httpReader.Close()
|
||||
}
|
||||
}()
|
||||
defer cancel()
|
||||
|
||||
// Used to verify if etag of object has changed since last read.
|
||||
var etag string
|
||||
|
||||
for req := range reqCh {
|
||||
// If this is the first request we may not need to do a getObject request yet.
|
||||
if req.isFirstReq {
|
||||
// First request is a Read/ReadAt.
|
||||
if req.isReadOp {
|
||||
// Differentiate between wanting the whole object and just a range.
|
||||
if req.isReadAt {
|
||||
// If this is a ReadAt request only get the specified range.
|
||||
// Range is set with respect to the offset and length of the buffer requested.
|
||||
// Do not set objectInfo from the first readAt request because it will not get
|
||||
// the whole object.
|
||||
opts.SetRange(req.Offset, req.Offset+int64(len(req.Buffer))-1)
|
||||
} else if req.Offset > 0 {
|
||||
opts.SetRange(req.Offset, 0)
|
||||
}
|
||||
httpReader, objectInfo, _, err = c.getObject(gctx, bucketName, objectName, opts)
|
||||
if err != nil {
|
||||
resCh <- getResponse{Error: err}
|
||||
return
|
||||
}
|
||||
etag = objectInfo.ETag
|
||||
// Read at least firstReq.Buffer bytes, if not we have
|
||||
// reached our EOF.
|
||||
size, err := readFull(httpReader, req.Buffer)
|
||||
totalRead += size
|
||||
if size > 0 && err == io.ErrUnexpectedEOF {
|
||||
if int64(size) < objectInfo.Size {
|
||||
// In situations when returned size
|
||||
// is less than the expected content
|
||||
// length set by the server, make sure
|
||||
// we return io.ErrUnexpectedEOF
|
||||
err = io.ErrUnexpectedEOF
|
||||
} else {
|
||||
// If an EOF happens after reading some but not
|
||||
// all the bytes ReadFull returns ErrUnexpectedEOF
|
||||
err = io.EOF
|
||||
}
|
||||
} else if size == 0 && err == io.EOF && objectInfo.Size > 0 {
|
||||
// Special cases when server writes more data
|
||||
// than the content-length, net/http response
|
||||
// body returns an error, instead of converting
|
||||
// it to io.EOF - return unexpected EOF.
|
||||
err = io.ErrUnexpectedEOF
|
||||
}
|
||||
// Send back the first response.
|
||||
resCh <- getResponse{
|
||||
objectInfo: objectInfo,
|
||||
Size: size,
|
||||
Error: err,
|
||||
didRead: true,
|
||||
}
|
||||
} else {
|
||||
// First request is a Stat or Seek call.
|
||||
// Only need to run a StatObject until an actual Read or ReadAt request comes through.
|
||||
|
||||
// Remove range header if already set, for stat Operations to get original file size.
|
||||
delete(opts.headers, "Range")
|
||||
objectInfo, err = c.StatObject(gctx, bucketName, objectName, StatObjectOptions(opts))
|
||||
if err != nil {
|
||||
resCh <- getResponse{
|
||||
Error: err,
|
||||
}
|
||||
// Exit the go-routine.
|
||||
return
|
||||
}
|
||||
etag = objectInfo.ETag
|
||||
// Send back the first response.
|
||||
resCh <- getResponse{
|
||||
objectInfo: objectInfo,
|
||||
}
|
||||
}
|
||||
} else if req.settingObjectInfo { // Request is just to get objectInfo.
|
||||
// Remove range header if already set, for stat Operations to get original file size.
|
||||
delete(opts.headers, "Range")
|
||||
// Check whether this is snowball
|
||||
// if yes do not use If-Match feature
|
||||
// it doesn't work.
|
||||
if etag != "" && !snowball {
|
||||
opts.SetMatchETag(etag)
|
||||
}
|
||||
objectInfo, err := c.StatObject(gctx, bucketName, objectName, StatObjectOptions(opts))
|
||||
if err != nil {
|
||||
resCh <- getResponse{
|
||||
Error: err,
|
||||
}
|
||||
// Exit the goroutine.
|
||||
return
|
||||
}
|
||||
// Send back the objectInfo.
|
||||
resCh <- getResponse{
|
||||
objectInfo: objectInfo,
|
||||
}
|
||||
} else {
|
||||
// Offset changes fetch the new object at an Offset.
|
||||
// Because the httpReader may not be set by the first
|
||||
// request if it was a stat or seek it must be checked
|
||||
// if the object has been read or not to only initialize
|
||||
// new ones when they haven't been already.
|
||||
// All readAt requests are new requests.
|
||||
if req.DidOffsetChange || !req.beenRead {
|
||||
// Check whether this is snowball
|
||||
// if yes do not use If-Match feature
|
||||
// it doesn't work.
|
||||
if etag != "" && !snowball {
|
||||
opts.SetMatchETag(etag)
|
||||
}
|
||||
if httpReader != nil {
|
||||
// Close previously opened http reader.
|
||||
httpReader.Close()
|
||||
}
|
||||
// If this request is a readAt only get the specified range.
|
||||
if req.isReadAt {
|
||||
// Range is set with respect to the offset and length of the buffer requested.
|
||||
opts.SetRange(req.Offset, req.Offset+int64(len(req.Buffer))-1)
|
||||
} else if req.Offset > 0 { // Range is set with respect to the offset.
|
||||
opts.SetRange(req.Offset, 0)
|
||||
} else {
|
||||
// Remove range header if already set
|
||||
delete(opts.headers, "Range")
|
||||
}
|
||||
httpReader, objectInfo, _, err = c.getObject(gctx, bucketName, objectName, opts)
|
||||
if err != nil {
|
||||
resCh <- getResponse{
|
||||
Error: err,
|
||||
}
|
||||
return
|
||||
}
|
||||
totalRead = 0
|
||||
}
|
||||
|
||||
// Read at least req.Buffer bytes, if not we have
|
||||
// reached our EOF.
|
||||
size, err := readFull(httpReader, req.Buffer)
|
||||
totalRead += size
|
||||
if size > 0 && err == io.ErrUnexpectedEOF {
|
||||
if int64(totalRead) < objectInfo.Size {
|
||||
// In situations when returned size
|
||||
// is less than the expected content
|
||||
// length set by the server, make sure
|
||||
// we return io.ErrUnexpectedEOF
|
||||
err = io.ErrUnexpectedEOF
|
||||
} else {
|
||||
// If an EOF happens after reading some but not
|
||||
// all the bytes ReadFull returns ErrUnexpectedEOF
|
||||
err = io.EOF
|
||||
}
|
||||
} else if size == 0 && err == io.EOF && objectInfo.Size > 0 {
|
||||
// Special cases when server writes more data
|
||||
// than the content-length, net/http response
|
||||
// body returns an error, instead of converting
|
||||
// it to io.EOF - return unexpected EOF.
|
||||
err = io.ErrUnexpectedEOF
|
||||
}
|
||||
|
||||
// Reply back how much was read.
|
||||
resCh <- getResponse{
|
||||
Size: size,
|
||||
Error: err,
|
||||
didRead: true,
|
||||
objectInfo: objectInfo,
|
||||
}
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
// Create a newObject through the information sent back by reqCh.
|
||||
return newObject(gctx, cancel, reqCh, resCh), nil
|
||||
}
|
||||
|
||||
// get request message container to communicate with internal
|
||||
// go-routine.
|
||||
type getRequest struct {
|
||||
Buffer []byte
|
||||
Offset int64 // readAt offset.
|
||||
DidOffsetChange bool // Tracks the offset changes for Seek requests.
|
||||
beenRead bool // Determines if this is the first time an object is being read.
|
||||
isReadAt bool // Determines if this request is a request to a specific range
|
||||
isReadOp bool // Determines if this request is a Read or Read/At request.
|
||||
isFirstReq bool // Determines if this request is the first time an object is being accessed.
|
||||
settingObjectInfo bool // Determines if this request is to set the objectInfo of an object.
|
||||
}
|
||||
|
||||
// get response message container to reply back for the request.
|
||||
type getResponse struct {
|
||||
Size int
|
||||
Error error
|
||||
didRead bool // Lets subsequent calls know whether or not httpReader has been initiated.
|
||||
objectInfo ObjectInfo // Used for the first request.
|
||||
}
|
||||
|
||||
// Object represents an open object. It implements
|
||||
// Reader, ReaderAt, Seeker, Closer for a HTTP stream.
|
||||
type Object struct {
|
||||
// Mutex.
|
||||
mutex *sync.Mutex
|
||||
|
||||
// User allocated and defined.
|
||||
reqCh chan<- getRequest
|
||||
resCh <-chan getResponse
|
||||
ctx context.Context
|
||||
cancel context.CancelFunc
|
||||
currOffset int64
|
||||
objectInfo ObjectInfo
|
||||
|
||||
// Ask lower level to initiate data fetching based on currOffset
|
||||
seekData bool
|
||||
|
||||
// Keeps track of closed call.
|
||||
isClosed bool
|
||||
|
||||
// Keeps track of if this is the first call.
|
||||
isStarted bool
|
||||
|
||||
// Previous error saved for future calls.
|
||||
prevErr error
|
||||
|
||||
// Keeps track of if this object has been read yet.
|
||||
beenRead bool
|
||||
|
||||
// Keeps track of if objectInfo has been set yet.
|
||||
objectInfoSet bool
|
||||
}
|
||||
|
||||
// doGetRequest - sends and blocks on the firstReqCh and reqCh of an object.
|
||||
// Returns back the size of the buffer read, if anything was read, as well
|
||||
// as any error encountered. For all first requests sent on the object
|
||||
// it is also responsible for sending back the objectInfo.
|
||||
func (o *Object) doGetRequest(request getRequest) (getResponse, error) {
|
||||
select {
|
||||
case <-o.ctx.Done():
|
||||
return getResponse{}, o.ctx.Err()
|
||||
case o.reqCh <- request:
|
||||
}
|
||||
|
||||
response := <-o.resCh
|
||||
|
||||
// Return any error to the top level.
|
||||
if response.Error != nil && response.Error != io.EOF {
|
||||
return response, response.Error
|
||||
}
|
||||
|
||||
// This was the first request.
|
||||
if !o.isStarted {
|
||||
// The object has been operated on.
|
||||
o.isStarted = true
|
||||
}
|
||||
// Set the objectInfo if the request was not readAt
|
||||
// and it hasn't been set before.
|
||||
if !o.objectInfoSet && !request.isReadAt {
|
||||
o.objectInfo = response.objectInfo
|
||||
o.objectInfoSet = true
|
||||
}
|
||||
// Set beenRead only if it has not been set before.
|
||||
if !o.beenRead {
|
||||
o.beenRead = response.didRead
|
||||
}
|
||||
// Data are ready on the wire, no need to reinitiate connection in lower level
|
||||
o.seekData = false
|
||||
|
||||
return response, response.Error
|
||||
}
|
||||
|
||||
// setOffset - handles the setting of offsets for
|
||||
// Read/ReadAt/Seek requests.
|
||||
func (o *Object) setOffset(bytesRead int64) error {
|
||||
// Update the currentOffset.
|
||||
o.currOffset += bytesRead
|
||||
|
||||
if o.objectInfo.Size > -1 && o.currOffset >= o.objectInfo.Size {
|
||||
return io.EOF
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Read reads up to len(b) bytes into b. It returns the number of
|
||||
// bytes read (0 <= n <= len(b)) and any error encountered. Returns
|
||||
// io.EOF upon end of file.
|
||||
func (o *Object) Read(b []byte) (n int, err error) {
|
||||
if o == nil {
|
||||
return 0, errInvalidArgument("Object is nil")
|
||||
}
|
||||
|
||||
// Locking.
|
||||
o.mutex.Lock()
|
||||
defer o.mutex.Unlock()
|
||||
|
||||
// prevErr is previous error saved from previous operation.
|
||||
if o.prevErr != nil || o.isClosed {
|
||||
return 0, o.prevErr
|
||||
}
|
||||
|
||||
// Create a new request.
|
||||
readReq := getRequest{
|
||||
isReadOp: true,
|
||||
beenRead: o.beenRead,
|
||||
Buffer: b,
|
||||
}
|
||||
|
||||
// Alert that this is the first request.
|
||||
if !o.isStarted {
|
||||
readReq.isFirstReq = true
|
||||
}
|
||||
|
||||
// Ask to establish a new data fetch routine based on seekData flag
|
||||
readReq.DidOffsetChange = o.seekData
|
||||
readReq.Offset = o.currOffset
|
||||
|
||||
// Send and receive from the first request.
|
||||
response, err := o.doGetRequest(readReq)
|
||||
if err != nil && err != io.EOF {
|
||||
// Save the error for future calls.
|
||||
o.prevErr = err
|
||||
return response.Size, err
|
||||
}
|
||||
|
||||
// Bytes read.
|
||||
bytesRead := int64(response.Size)
|
||||
|
||||
// Set the new offset.
|
||||
oerr := o.setOffset(bytesRead)
|
||||
if oerr != nil {
|
||||
// Save the error for future calls.
|
||||
o.prevErr = oerr
|
||||
return response.Size, oerr
|
||||
}
|
||||
|
||||
// Return the response.
|
||||
return response.Size, err
|
||||
}
|
||||
|
||||
// Stat returns the ObjectInfo structure describing Object.
|
||||
func (o *Object) Stat() (ObjectInfo, error) {
|
||||
if o == nil {
|
||||
return ObjectInfo{}, errInvalidArgument("Object is nil")
|
||||
}
|
||||
// Locking.
|
||||
o.mutex.Lock()
|
||||
defer o.mutex.Unlock()
|
||||
|
||||
if o.prevErr != nil && o.prevErr != io.EOF || o.isClosed {
|
||||
return ObjectInfo{}, o.prevErr
|
||||
}
|
||||
|
||||
// This is the first request.
|
||||
if !o.isStarted || !o.objectInfoSet {
|
||||
// Send the request and get the response.
|
||||
_, err := o.doGetRequest(getRequest{
|
||||
isFirstReq: !o.isStarted,
|
||||
settingObjectInfo: !o.objectInfoSet,
|
||||
})
|
||||
if err != nil {
|
||||
o.prevErr = err
|
||||
return ObjectInfo{}, err
|
||||
}
|
||||
}
|
||||
|
||||
return o.objectInfo, nil
|
||||
}
|
||||
|
||||
// ReadAt reads len(b) bytes from the File starting at byte offset
|
||||
// off. It returns the number of bytes read and the error, if any.
|
||||
// ReadAt always returns a non-nil error when n < len(b). At end of
|
||||
// file, that error is io.EOF.
|
||||
func (o *Object) ReadAt(b []byte, offset int64) (n int, err error) {
|
||||
if o == nil {
|
||||
return 0, errInvalidArgument("Object is nil")
|
||||
}
|
||||
|
||||
// Locking.
|
||||
o.mutex.Lock()
|
||||
defer o.mutex.Unlock()
|
||||
|
||||
// prevErr is error which was saved in previous operation.
|
||||
if o.prevErr != nil && o.prevErr != io.EOF || o.isClosed {
|
||||
return 0, o.prevErr
|
||||
}
|
||||
|
||||
// Set the current offset to ReadAt offset, because the current offset will be shifted at the end of this method.
|
||||
o.currOffset = offset
|
||||
|
||||
// Can only compare offsets to size when size has been set.
|
||||
if o.objectInfoSet {
|
||||
// If offset is negative than we return io.EOF.
|
||||
// If offset is greater than or equal to object size we return io.EOF.
|
||||
if (o.objectInfo.Size > -1 && offset >= o.objectInfo.Size) || offset < 0 {
|
||||
return 0, io.EOF
|
||||
}
|
||||
}
|
||||
|
||||
// Create the new readAt request.
|
||||
readAtReq := getRequest{
|
||||
isReadOp: true,
|
||||
isReadAt: true,
|
||||
DidOffsetChange: true, // Offset always changes.
|
||||
beenRead: o.beenRead, // Set if this is the first request to try and read.
|
||||
Offset: offset, // Set the offset.
|
||||
Buffer: b,
|
||||
}
|
||||
|
||||
// Alert that this is the first request.
|
||||
if !o.isStarted {
|
||||
readAtReq.isFirstReq = true
|
||||
}
|
||||
|
||||
// Send and receive from the first request.
|
||||
response, err := o.doGetRequest(readAtReq)
|
||||
if err != nil && err != io.EOF {
|
||||
// Save the error.
|
||||
o.prevErr = err
|
||||
return response.Size, err
|
||||
}
|
||||
// Bytes read.
|
||||
bytesRead := int64(response.Size)
|
||||
// There is no valid objectInfo yet
|
||||
// to compare against for EOF.
|
||||
if !o.objectInfoSet {
|
||||
// Update the currentOffset.
|
||||
o.currOffset += bytesRead
|
||||
} else {
|
||||
// If this was not the first request update
|
||||
// the offsets and compare against objectInfo
|
||||
// for EOF.
|
||||
oerr := o.setOffset(bytesRead)
|
||||
if oerr != nil {
|
||||
o.prevErr = oerr
|
||||
return response.Size, oerr
|
||||
}
|
||||
}
|
||||
return response.Size, err
|
||||
}
|
||||
|
||||
// Seek sets the offset for the next Read or Write to offset,
|
||||
// interpreted according to whence: 0 means relative to the
|
||||
// origin of the file, 1 means relative to the current offset,
|
||||
// and 2 means relative to the end.
|
||||
// Seek returns the new offset and an error, if any.
|
||||
//
|
||||
// Seeking to a negative offset is an error. Seeking to any positive
|
||||
// offset is legal, subsequent io operations succeed until the
|
||||
// underlying object is not closed.
|
||||
func (o *Object) Seek(offset int64, whence int) (n int64, err error) {
|
||||
if o == nil {
|
||||
return 0, errInvalidArgument("Object is nil")
|
||||
}
|
||||
|
||||
// Locking.
|
||||
o.mutex.Lock()
|
||||
defer o.mutex.Unlock()
|
||||
|
||||
// At EOF seeking is legal allow only io.EOF, for any other errors we return.
|
||||
if o.prevErr != nil && o.prevErr != io.EOF {
|
||||
return 0, o.prevErr
|
||||
}
|
||||
|
||||
// Negative offset is valid for whence of '2'.
|
||||
if offset < 0 && whence != 2 {
|
||||
return 0, errInvalidArgument(fmt.Sprintf("Negative position not allowed for %d", whence))
|
||||
}
|
||||
|
||||
// This is the first request. So before anything else
|
||||
// get the ObjectInfo.
|
||||
if !o.isStarted || !o.objectInfoSet {
|
||||
// Create the new Seek request.
|
||||
seekReq := getRequest{
|
||||
isReadOp: false,
|
||||
Offset: offset,
|
||||
isFirstReq: true,
|
||||
}
|
||||
// Send and receive from the seek request.
|
||||
_, err := o.doGetRequest(seekReq)
|
||||
if err != nil {
|
||||
// Save the error.
|
||||
o.prevErr = err
|
||||
return 0, err
|
||||
}
|
||||
}
|
||||
|
||||
newOffset := o.currOffset
|
||||
|
||||
// Switch through whence.
|
||||
switch whence {
|
||||
default:
|
||||
return 0, errInvalidArgument(fmt.Sprintf("Invalid whence %d", whence))
|
||||
case 0:
|
||||
if o.objectInfo.Size > -1 && offset > o.objectInfo.Size {
|
||||
return 0, io.EOF
|
||||
}
|
||||
newOffset = offset
|
||||
case 1:
|
||||
if o.objectInfo.Size > -1 && o.currOffset+offset > o.objectInfo.Size {
|
||||
return 0, io.EOF
|
||||
}
|
||||
newOffset += offset
|
||||
case 2:
|
||||
// If we don't know the object size return an error for io.SeekEnd
|
||||
if o.objectInfo.Size < 0 {
|
||||
return 0, errInvalidArgument("Whence END is not supported when the object size is unknown")
|
||||
}
|
||||
// Seeking to positive offset is valid for whence '2', but
|
||||
// since we are backing a Reader we have reached 'EOF' if
|
||||
// offset is positive.
|
||||
if offset > 0 {
|
||||
return 0, io.EOF
|
||||
}
|
||||
// Seeking to negative position not allowed for whence.
|
||||
if o.objectInfo.Size+offset < 0 {
|
||||
return 0, errInvalidArgument(fmt.Sprintf("Seeking at negative offset not allowed for %d", whence))
|
||||
}
|
||||
newOffset = o.objectInfo.Size + offset
|
||||
}
|
||||
// Reset the saved error since we successfully seeked, let the Read
|
||||
// and ReadAt decide.
|
||||
if o.prevErr == io.EOF {
|
||||
o.prevErr = nil
|
||||
}
|
||||
|
||||
// Ask lower level to fetch again from source when necessary
|
||||
o.seekData = (newOffset != o.currOffset) || o.seekData
|
||||
o.currOffset = newOffset
|
||||
|
||||
// Return the effective offset.
|
||||
return o.currOffset, nil
|
||||
}
|
||||
|
||||
// Close - The behavior of Close after the first call returns error
|
||||
// for subsequent Close() calls.
|
||||
func (o *Object) Close() (err error) {
|
||||
if o == nil {
|
||||
return errInvalidArgument("Object is nil")
|
||||
}
|
||||
|
||||
// Locking.
|
||||
o.mutex.Lock()
|
||||
defer o.mutex.Unlock()
|
||||
|
||||
// if already closed return an error.
|
||||
if o.isClosed {
|
||||
return o.prevErr
|
||||
}
|
||||
|
||||
// Close successfully.
|
||||
o.cancel()
|
||||
|
||||
// Close the request channel to indicate the internal go-routine to exit.
|
||||
close(o.reqCh)
|
||||
|
||||
// Save for future operations.
|
||||
errMsg := "Object is already closed. Bad file descriptor."
|
||||
o.prevErr = errors.New(errMsg)
|
||||
// Save here that we closed done channel successfully.
|
||||
o.isClosed = true
|
||||
return nil
|
||||
}
|
||||
|
||||
// newObject instantiates a new *minio.Object*
|
||||
// ObjectInfo will be set by setObjectInfo
|
||||
func newObject(ctx context.Context, cancel context.CancelFunc, reqCh chan<- getRequest, resCh <-chan getResponse) *Object {
|
||||
return &Object{
|
||||
ctx: ctx,
|
||||
cancel: cancel,
|
||||
mutex: &sync.Mutex{},
|
||||
reqCh: reqCh,
|
||||
resCh: resCh,
|
||||
}
|
||||
}
|
||||
|
||||
// getObject - retrieve object from Object Storage.
|
||||
//
|
||||
// Additionally this function also takes range arguments to download the specified
|
||||
// range bytes of an object. Setting offset and length = 0 will download the full object.
|
||||
//
|
||||
// For more information about the HTTP Range header.
|
||||
// go to http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35.
|
||||
func (c *Client) getObject(ctx context.Context, bucketName, objectName string, opts GetObjectOptions) (io.ReadCloser, ObjectInfo, http.Header, error) {
|
||||
// Validate input arguments.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return nil, ObjectInfo{}, nil, ErrorResponse{
|
||||
StatusCode: http.StatusBadRequest,
|
||||
Code: "InvalidBucketName",
|
||||
Message: err.Error(),
|
||||
}
|
||||
}
|
||||
if err := s3utils.CheckValidObjectName(objectName); err != nil {
|
||||
return nil, ObjectInfo{}, nil, ErrorResponse{
|
||||
StatusCode: http.StatusBadRequest,
|
||||
Code: "XMinioInvalidObjectName",
|
||||
Message: err.Error(),
|
||||
}
|
||||
}
|
||||
|
||||
// Execute GET on objectName.
|
||||
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
objectName: objectName,
|
||||
queryValues: opts.toQueryValues(),
|
||||
customHeader: opts.Header(),
|
||||
contentSHA256Hex: emptySHA256Hex,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, ObjectInfo{}, nil, err
|
||||
}
|
||||
if resp != nil {
|
||||
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusPartialContent {
|
||||
return nil, ObjectInfo{}, nil, httpRespToErrorResponse(resp, bucketName, objectName)
|
||||
}
|
||||
}
|
||||
|
||||
objectStat, err := ToObjectInfo(bucketName, objectName, resp.Header)
|
||||
if err != nil {
|
||||
closeResponse(resp)
|
||||
return nil, ObjectInfo{}, nil, err
|
||||
}
|
||||
|
||||
// do not close body here, caller will close
|
||||
return resp.Body, objectStat, resp.Header, nil
|
||||
}
|
||||
+203
@@ -0,0 +1,203 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2015-2020 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"github.com/minio/minio-go/v7/pkg/encrypt"
|
||||
)
|
||||
|
||||
// AdvancedGetOptions for internal use by MinIO server - not intended for client use.
|
||||
type AdvancedGetOptions struct {
|
||||
ReplicationDeleteMarker bool
|
||||
IsReplicationReadyForDeleteMarker bool
|
||||
ReplicationProxyRequest string
|
||||
}
|
||||
|
||||
// GetObjectOptions are used to specify additional headers or options
|
||||
// during GET requests.
|
||||
type GetObjectOptions struct {
|
||||
headers map[string]string
|
||||
reqParams url.Values
|
||||
ServerSideEncryption encrypt.ServerSide
|
||||
VersionID string
|
||||
PartNumber int
|
||||
|
||||
// Include any checksums, if object was uploaded with checksum.
|
||||
// For multipart objects this is a checksum of part checksums.
|
||||
// https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html
|
||||
Checksum bool
|
||||
|
||||
// To be not used by external applications
|
||||
Internal AdvancedGetOptions
|
||||
}
|
||||
|
||||
// StatObjectOptions are used to specify additional headers or options
|
||||
// during GET info/stat requests.
|
||||
type StatObjectOptions = GetObjectOptions
|
||||
|
||||
// Header returns the http.Header representation of the GET options.
|
||||
func (o GetObjectOptions) Header() http.Header {
|
||||
headers := make(http.Header, len(o.headers))
|
||||
for k, v := range o.headers {
|
||||
headers.Set(k, v)
|
||||
}
|
||||
if o.ServerSideEncryption != nil && o.ServerSideEncryption.Type() == encrypt.SSEC {
|
||||
o.ServerSideEncryption.Marshal(headers)
|
||||
}
|
||||
// this header is set for active-active replication scenario where GET/HEAD
|
||||
// to site A is proxy'd to site B if object/version missing on site A.
|
||||
if o.Internal.ReplicationProxyRequest != "" {
|
||||
headers.Set(minIOBucketReplicationProxyRequest, o.Internal.ReplicationProxyRequest)
|
||||
}
|
||||
if o.Checksum {
|
||||
headers.Set("x-amz-checksum-mode", "ENABLED")
|
||||
}
|
||||
return headers
|
||||
}
|
||||
|
||||
// Set adds a key value pair to the options. The
|
||||
// key-value pair will be part of the HTTP GET request
|
||||
// headers.
|
||||
func (o *GetObjectOptions) Set(key, value string) {
|
||||
if o.headers == nil {
|
||||
o.headers = make(map[string]string)
|
||||
}
|
||||
o.headers[http.CanonicalHeaderKey(key)] = value
|
||||
}
|
||||
|
||||
// SetReqParam - set request query string parameter
|
||||
// supported key: see supportedQueryValues and allowedCustomQueryPrefix.
|
||||
// If an unsupported key is passed in, it will be ignored and nothing will be done.
|
||||
func (o *GetObjectOptions) SetReqParam(key, value string) {
|
||||
if !isCustomQueryValue(key) && !isStandardQueryValue(key) {
|
||||
// do nothing
|
||||
return
|
||||
}
|
||||
if o.reqParams == nil {
|
||||
o.reqParams = make(url.Values)
|
||||
}
|
||||
o.reqParams.Set(key, value)
|
||||
}
|
||||
|
||||
// AddReqParam - add request query string parameter
|
||||
// supported key: see supportedQueryValues and allowedCustomQueryPrefix.
|
||||
// If an unsupported key is passed in, it will be ignored and nothing will be done.
|
||||
func (o *GetObjectOptions) AddReqParam(key, value string) {
|
||||
if !isCustomQueryValue(key) && !isStandardQueryValue(key) {
|
||||
// do nothing
|
||||
return
|
||||
}
|
||||
if o.reqParams == nil {
|
||||
o.reqParams = make(url.Values)
|
||||
}
|
||||
o.reqParams.Add(key, value)
|
||||
}
|
||||
|
||||
// SetMatchETag - set match etag.
|
||||
func (o *GetObjectOptions) SetMatchETag(etag string) error {
|
||||
if etag == "" {
|
||||
return errInvalidArgument("ETag cannot be empty.")
|
||||
}
|
||||
o.Set("If-Match", "\""+etag+"\"")
|
||||
return nil
|
||||
}
|
||||
|
||||
// SetMatchETagExcept - set match etag except.
|
||||
func (o *GetObjectOptions) SetMatchETagExcept(etag string) error {
|
||||
if etag == "" {
|
||||
return errInvalidArgument("ETag cannot be empty.")
|
||||
}
|
||||
o.Set("If-None-Match", "\""+etag+"\"")
|
||||
return nil
|
||||
}
|
||||
|
||||
// SetUnmodified - set unmodified time since.
|
||||
func (o *GetObjectOptions) SetUnmodified(modTime time.Time) error {
|
||||
if modTime.IsZero() {
|
||||
return errInvalidArgument("Modified since cannot be empty.")
|
||||
}
|
||||
o.Set("If-Unmodified-Since", modTime.Format(http.TimeFormat))
|
||||
return nil
|
||||
}
|
||||
|
||||
// SetModified - set modified time since.
|
||||
func (o *GetObjectOptions) SetModified(modTime time.Time) error {
|
||||
if modTime.IsZero() {
|
||||
return errInvalidArgument("Modified since cannot be empty.")
|
||||
}
|
||||
o.Set("If-Modified-Since", modTime.Format(http.TimeFormat))
|
||||
return nil
|
||||
}
|
||||
|
||||
// SetRange - set the start and end offset of the object to be read.
|
||||
// See https://tools.ietf.org/html/rfc7233#section-3.1 for reference.
|
||||
func (o *GetObjectOptions) SetRange(start, end int64) error {
|
||||
switch {
|
||||
case start == 0 && end < 0:
|
||||
// Read last '-end' bytes. `bytes=-N`.
|
||||
o.Set("Range", fmt.Sprintf("bytes=%d", end))
|
||||
case 0 < start && end == 0:
|
||||
// Read everything starting from offset
|
||||
// 'start'. `bytes=N-`.
|
||||
o.Set("Range", fmt.Sprintf("bytes=%d-", start))
|
||||
case 0 <= start && start <= end:
|
||||
// Read everything starting at 'start' till the
|
||||
// 'end'. `bytes=N-M`
|
||||
o.Set("Range", fmt.Sprintf("bytes=%d-%d", start, end))
|
||||
default:
|
||||
// All other cases such as
|
||||
// bytes=-3-
|
||||
// bytes=5-3
|
||||
// bytes=-2-4
|
||||
// bytes=-3-0
|
||||
// bytes=-3--2
|
||||
// are invalid.
|
||||
return errInvalidArgument(
|
||||
fmt.Sprintf(
|
||||
"Invalid range specified: start=%d end=%d",
|
||||
start, end))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// toQueryValues - Convert the versionId, partNumber, and reqParams in Options to query string parameters.
|
||||
func (o *GetObjectOptions) toQueryValues() url.Values {
|
||||
urlValues := make(url.Values)
|
||||
if o.VersionID != "" {
|
||||
urlValues.Set("versionId", o.VersionID)
|
||||
}
|
||||
if o.PartNumber > 0 {
|
||||
urlValues.Set("partNumber", strconv.Itoa(o.PartNumber))
|
||||
}
|
||||
|
||||
if o.reqParams != nil {
|
||||
for key, values := range o.reqParams {
|
||||
for _, value := range values {
|
||||
urlValues.Add(key, value)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return urlValues
|
||||
}
|
||||
+1057
File diff suppressed because it is too large
Load Diff
+176
@@ -0,0 +1,176 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2020 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/xml"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/url"
|
||||
|
||||
"github.com/minio/minio-go/v7/pkg/s3utils"
|
||||
)
|
||||
|
||||
// objectLegalHold - object legal hold specified in
|
||||
// https://docs.aws.amazon.com/AmazonS3/latest/API/archive-RESTObjectPUTLegalHold.html
|
||||
type objectLegalHold struct {
|
||||
XMLNS string `xml:"xmlns,attr,omitempty"`
|
||||
XMLName xml.Name `xml:"LegalHold"`
|
||||
Status LegalHoldStatus `xml:"Status,omitempty"`
|
||||
}
|
||||
|
||||
// PutObjectLegalHoldOptions represents options specified by user for PutObjectLegalHold call
|
||||
type PutObjectLegalHoldOptions struct {
|
||||
VersionID string
|
||||
Status *LegalHoldStatus
|
||||
}
|
||||
|
||||
// GetObjectLegalHoldOptions represents options specified by user for GetObjectLegalHold call
|
||||
type GetObjectLegalHoldOptions struct {
|
||||
VersionID string
|
||||
}
|
||||
|
||||
// LegalHoldStatus - object legal hold status.
|
||||
type LegalHoldStatus string
|
||||
|
||||
const (
|
||||
// LegalHoldEnabled indicates legal hold is enabled
|
||||
LegalHoldEnabled LegalHoldStatus = "ON"
|
||||
|
||||
// LegalHoldDisabled indicates legal hold is disabled
|
||||
LegalHoldDisabled LegalHoldStatus = "OFF"
|
||||
)
|
||||
|
||||
func (r LegalHoldStatus) String() string {
|
||||
return string(r)
|
||||
}
|
||||
|
||||
// IsValid - check whether this legal hold status is valid or not.
|
||||
func (r LegalHoldStatus) IsValid() bool {
|
||||
return r == LegalHoldEnabled || r == LegalHoldDisabled
|
||||
}
|
||||
|
||||
func newObjectLegalHold(status *LegalHoldStatus) (*objectLegalHold, error) {
|
||||
if status == nil {
|
||||
return nil, fmt.Errorf("Status not set")
|
||||
}
|
||||
if !status.IsValid() {
|
||||
return nil, fmt.Errorf("invalid legal hold status `%v`", status)
|
||||
}
|
||||
legalHold := &objectLegalHold{
|
||||
Status: *status,
|
||||
}
|
||||
return legalHold, nil
|
||||
}
|
||||
|
||||
// PutObjectLegalHold : sets object legal hold for a given object and versionID.
|
||||
func (c *Client) PutObjectLegalHold(ctx context.Context, bucketName, objectName string, opts PutObjectLegalHoldOptions) error {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if err := s3utils.CheckValidObjectName(objectName); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Get resources properly escaped and lined up before
|
||||
// using them in http request.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("legal-hold", "")
|
||||
|
||||
if opts.VersionID != "" {
|
||||
urlValues.Set("versionId", opts.VersionID)
|
||||
}
|
||||
|
||||
lh, err := newObjectLegalHold(opts.Status)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
lhData, err := xml.Marshal(lh)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
reqMetadata := requestMetadata{
|
||||
bucketName: bucketName,
|
||||
objectName: objectName,
|
||||
queryValues: urlValues,
|
||||
contentBody: bytes.NewReader(lhData),
|
||||
contentLength: int64(len(lhData)),
|
||||
contentMD5Base64: sumMD5Base64(lhData),
|
||||
contentSHA256Hex: sum256Hex(lhData),
|
||||
}
|
||||
|
||||
// Execute PUT Object Legal Hold.
|
||||
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if resp != nil {
|
||||
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusNoContent {
|
||||
return httpRespToErrorResponse(resp, bucketName, objectName)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetObjectLegalHold gets legal-hold status of given object.
|
||||
func (c *Client) GetObjectLegalHold(ctx context.Context, bucketName, objectName string, opts GetObjectLegalHoldOptions) (status *LegalHoldStatus, err error) {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if err := s3utils.CheckValidObjectName(objectName); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("legal-hold", "")
|
||||
|
||||
if opts.VersionID != "" {
|
||||
urlValues.Set("versionId", opts.VersionID)
|
||||
}
|
||||
|
||||
// Execute GET on bucket to list objects.
|
||||
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
objectName: objectName,
|
||||
queryValues: urlValues,
|
||||
contentSHA256Hex: emptySHA256Hex,
|
||||
})
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if resp != nil {
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return nil, httpRespToErrorResponse(resp, bucketName, objectName)
|
||||
}
|
||||
}
|
||||
lh := &objectLegalHold{}
|
||||
if err = xml.NewDecoder(resp.Body).Decode(lh); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &lh.Status, nil
|
||||
}
|
||||
+241
@@ -0,0 +1,241 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2019 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/xml"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"time"
|
||||
|
||||
"github.com/minio/minio-go/v7/pkg/s3utils"
|
||||
)
|
||||
|
||||
// RetentionMode - object retention mode.
|
||||
type RetentionMode string
|
||||
|
||||
const (
|
||||
// Governance - governance mode.
|
||||
Governance RetentionMode = "GOVERNANCE"
|
||||
|
||||
// Compliance - compliance mode.
|
||||
Compliance RetentionMode = "COMPLIANCE"
|
||||
)
|
||||
|
||||
func (r RetentionMode) String() string {
|
||||
return string(r)
|
||||
}
|
||||
|
||||
// IsValid - check whether this retention mode is valid or not.
|
||||
func (r RetentionMode) IsValid() bool {
|
||||
return r == Governance || r == Compliance
|
||||
}
|
||||
|
||||
// ValidityUnit - retention validity unit.
|
||||
type ValidityUnit string
|
||||
|
||||
const (
|
||||
// Days - denotes no. of days.
|
||||
Days ValidityUnit = "DAYS"
|
||||
|
||||
// Years - denotes no. of years.
|
||||
Years ValidityUnit = "YEARS"
|
||||
)
|
||||
|
||||
func (unit ValidityUnit) String() string {
|
||||
return string(unit)
|
||||
}
|
||||
|
||||
// IsValid - check whether this validity unit is valid or not.
|
||||
func (unit ValidityUnit) isValid() bool {
|
||||
return unit == Days || unit == Years
|
||||
}
|
||||
|
||||
// Retention - bucket level retention configuration.
|
||||
type Retention struct {
|
||||
Mode RetentionMode
|
||||
Validity time.Duration
|
||||
}
|
||||
|
||||
func (r Retention) String() string {
|
||||
return fmt.Sprintf("{Mode:%v, Validity:%v}", r.Mode, r.Validity)
|
||||
}
|
||||
|
||||
// IsEmpty - returns whether retention is empty or not.
|
||||
func (r Retention) IsEmpty() bool {
|
||||
return r.Mode == "" || r.Validity == 0
|
||||
}
|
||||
|
||||
// objectLockConfig - object lock configuration specified in
|
||||
// https://docs.aws.amazon.com/AmazonS3/latest/API/Type_API_ObjectLockConfiguration.html
|
||||
type objectLockConfig struct {
|
||||
XMLNS string `xml:"xmlns,attr,omitempty"`
|
||||
XMLName xml.Name `xml:"ObjectLockConfiguration"`
|
||||
ObjectLockEnabled string `xml:"ObjectLockEnabled"`
|
||||
Rule *struct {
|
||||
DefaultRetention struct {
|
||||
Mode RetentionMode `xml:"Mode"`
|
||||
Days *uint `xml:"Days"`
|
||||
Years *uint `xml:"Years"`
|
||||
} `xml:"DefaultRetention"`
|
||||
} `xml:"Rule,omitempty"`
|
||||
}
|
||||
|
||||
func newObjectLockConfig(mode *RetentionMode, validity *uint, unit *ValidityUnit) (*objectLockConfig, error) {
|
||||
config := &objectLockConfig{
|
||||
ObjectLockEnabled: "Enabled",
|
||||
}
|
||||
|
||||
if mode != nil && validity != nil && unit != nil {
|
||||
if !mode.IsValid() {
|
||||
return nil, fmt.Errorf("invalid retention mode `%v`", mode)
|
||||
}
|
||||
|
||||
if !unit.isValid() {
|
||||
return nil, fmt.Errorf("invalid validity unit `%v`", unit)
|
||||
}
|
||||
|
||||
config.Rule = &struct {
|
||||
DefaultRetention struct {
|
||||
Mode RetentionMode `xml:"Mode"`
|
||||
Days *uint `xml:"Days"`
|
||||
Years *uint `xml:"Years"`
|
||||
} `xml:"DefaultRetention"`
|
||||
}{}
|
||||
|
||||
config.Rule.DefaultRetention.Mode = *mode
|
||||
if *unit == Days {
|
||||
config.Rule.DefaultRetention.Days = validity
|
||||
} else {
|
||||
config.Rule.DefaultRetention.Years = validity
|
||||
}
|
||||
|
||||
return config, nil
|
||||
}
|
||||
|
||||
if mode == nil && validity == nil && unit == nil {
|
||||
return config, nil
|
||||
}
|
||||
|
||||
return nil, fmt.Errorf("all of retention mode, validity and validity unit must be passed")
|
||||
}
|
||||
|
||||
// SetBucketObjectLockConfig sets object lock configuration in given bucket. mode, validity and unit are either all set or all nil.
|
||||
func (c *Client) SetBucketObjectLockConfig(ctx context.Context, bucketName string, mode *RetentionMode, validity *uint, unit *ValidityUnit) error {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Get resources properly escaped and lined up before
|
||||
// using them in http request.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("object-lock", "")
|
||||
|
||||
config, err := newObjectLockConfig(mode, validity, unit)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
configData, err := xml.Marshal(config)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
reqMetadata := requestMetadata{
|
||||
bucketName: bucketName,
|
||||
queryValues: urlValues,
|
||||
contentBody: bytes.NewReader(configData),
|
||||
contentLength: int64(len(configData)),
|
||||
contentMD5Base64: sumMD5Base64(configData),
|
||||
contentSHA256Hex: sum256Hex(configData),
|
||||
}
|
||||
|
||||
// Execute PUT bucket object lock configuration.
|
||||
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if resp != nil {
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetObjectLockConfig gets object lock configuration of given bucket.
|
||||
func (c *Client) GetObjectLockConfig(ctx context.Context, bucketName string) (objectLock string, mode *RetentionMode, validity *uint, unit *ValidityUnit, err error) {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return "", nil, nil, nil, err
|
||||
}
|
||||
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("object-lock", "")
|
||||
|
||||
// Execute GET on bucket to list objects.
|
||||
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
queryValues: urlValues,
|
||||
contentSHA256Hex: emptySHA256Hex,
|
||||
})
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return "", nil, nil, nil, err
|
||||
}
|
||||
if resp != nil {
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return "", nil, nil, nil, httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
}
|
||||
config := &objectLockConfig{}
|
||||
if err = xml.NewDecoder(resp.Body).Decode(config); err != nil {
|
||||
return "", nil, nil, nil, err
|
||||
}
|
||||
|
||||
if config.Rule != nil {
|
||||
mode = &config.Rule.DefaultRetention.Mode
|
||||
if config.Rule.DefaultRetention.Days != nil {
|
||||
validity = config.Rule.DefaultRetention.Days
|
||||
days := Days
|
||||
unit = &days
|
||||
} else {
|
||||
validity = config.Rule.DefaultRetention.Years
|
||||
years := Years
|
||||
unit = &years
|
||||
}
|
||||
return config.ObjectLockEnabled, mode, validity, unit, nil
|
||||
}
|
||||
return config.ObjectLockEnabled, nil, nil, nil, nil
|
||||
}
|
||||
|
||||
// GetBucketObjectLockConfig gets object lock configuration of given bucket.
|
||||
func (c *Client) GetBucketObjectLockConfig(ctx context.Context, bucketName string) (mode *RetentionMode, validity *uint, unit *ValidityUnit, err error) {
|
||||
_, mode, validity, unit, err = c.GetObjectLockConfig(ctx, bucketName)
|
||||
return mode, validity, unit, err
|
||||
}
|
||||
|
||||
// SetObjectLockConfig sets object lock configuration in given bucket. mode, validity and unit are either all set or all nil.
|
||||
func (c *Client) SetObjectLockConfig(ctx context.Context, bucketName string, mode *RetentionMode, validity *uint, unit *ValidityUnit) error {
|
||||
return c.SetBucketObjectLockConfig(ctx, bucketName, mode, validity, unit)
|
||||
}
|
||||
+165
@@ -0,0 +1,165 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2019-2020 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/xml"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"time"
|
||||
|
||||
"github.com/minio/minio-go/v7/pkg/s3utils"
|
||||
)
|
||||
|
||||
// objectRetention - object retention specified in
|
||||
// https://docs.aws.amazon.com/AmazonS3/latest/API/Type_API_ObjectLockConfiguration.html
|
||||
type objectRetention struct {
|
||||
XMLNS string `xml:"xmlns,attr,omitempty"`
|
||||
XMLName xml.Name `xml:"Retention"`
|
||||
Mode RetentionMode `xml:"Mode,omitempty"`
|
||||
RetainUntilDate *time.Time `type:"timestamp" timestampFormat:"iso8601" xml:"RetainUntilDate,omitempty"`
|
||||
}
|
||||
|
||||
func newObjectRetention(mode *RetentionMode, date *time.Time) (*objectRetention, error) {
|
||||
objectRetention := &objectRetention{}
|
||||
|
||||
if date != nil && !date.IsZero() {
|
||||
objectRetention.RetainUntilDate = date
|
||||
}
|
||||
if mode != nil {
|
||||
if !mode.IsValid() {
|
||||
return nil, fmt.Errorf("invalid retention mode `%v`", mode)
|
||||
}
|
||||
objectRetention.Mode = *mode
|
||||
}
|
||||
|
||||
return objectRetention, nil
|
||||
}
|
||||
|
||||
// PutObjectRetentionOptions represents options specified by user for PutObject call
|
||||
type PutObjectRetentionOptions struct {
|
||||
GovernanceBypass bool
|
||||
Mode *RetentionMode
|
||||
RetainUntilDate *time.Time
|
||||
VersionID string
|
||||
}
|
||||
|
||||
// PutObjectRetention sets object retention for a given object and versionID.
|
||||
func (c *Client) PutObjectRetention(ctx context.Context, bucketName, objectName string, opts PutObjectRetentionOptions) error {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if err := s3utils.CheckValidObjectName(objectName); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Get resources properly escaped and lined up before
|
||||
// using them in http request.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("retention", "")
|
||||
|
||||
if opts.VersionID != "" {
|
||||
urlValues.Set("versionId", opts.VersionID)
|
||||
}
|
||||
|
||||
retention, err := newObjectRetention(opts.Mode, opts.RetainUntilDate)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
retentionData, err := xml.Marshal(retention)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Build headers.
|
||||
headers := make(http.Header)
|
||||
|
||||
if opts.GovernanceBypass {
|
||||
// Set the bypass goverenance retention header
|
||||
headers.Set(amzBypassGovernance, "true")
|
||||
}
|
||||
|
||||
reqMetadata := requestMetadata{
|
||||
bucketName: bucketName,
|
||||
objectName: objectName,
|
||||
queryValues: urlValues,
|
||||
contentBody: bytes.NewReader(retentionData),
|
||||
contentLength: int64(len(retentionData)),
|
||||
contentMD5Base64: sumMD5Base64(retentionData),
|
||||
contentSHA256Hex: sum256Hex(retentionData),
|
||||
customHeader: headers,
|
||||
}
|
||||
|
||||
// Execute PUT Object Retention.
|
||||
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if resp != nil {
|
||||
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusNoContent {
|
||||
return httpRespToErrorResponse(resp, bucketName, objectName)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetObjectRetention gets retention of given object.
|
||||
func (c *Client) GetObjectRetention(ctx context.Context, bucketName, objectName, versionID string) (mode *RetentionMode, retainUntilDate *time.Time, err error) {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
|
||||
if err := s3utils.CheckValidObjectName(objectName); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("retention", "")
|
||||
if versionID != "" {
|
||||
urlValues.Set("versionId", versionID)
|
||||
}
|
||||
// Execute GET on bucket to list objects.
|
||||
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
objectName: objectName,
|
||||
queryValues: urlValues,
|
||||
contentSHA256Hex: emptySHA256Hex,
|
||||
})
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
if resp != nil {
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return nil, nil, httpRespToErrorResponse(resp, bucketName, objectName)
|
||||
}
|
||||
}
|
||||
retention := &objectRetention{}
|
||||
if err = xml.NewDecoder(resp.Body).Decode(retention); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
|
||||
return &retention.Mode, retention.RetainUntilDate, nil
|
||||
}
|
||||
+177
@@ -0,0 +1,177 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2020 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/xml"
|
||||
"net/http"
|
||||
"net/url"
|
||||
|
||||
"github.com/minio/minio-go/v7/pkg/s3utils"
|
||||
"github.com/minio/minio-go/v7/pkg/tags"
|
||||
)
|
||||
|
||||
// PutObjectTaggingOptions holds an object version id
|
||||
// to update tag(s) of a specific object version
|
||||
type PutObjectTaggingOptions struct {
|
||||
VersionID string
|
||||
Internal AdvancedObjectTaggingOptions
|
||||
}
|
||||
|
||||
// AdvancedObjectTaggingOptions for internal use by MinIO server - not intended for client use.
|
||||
type AdvancedObjectTaggingOptions struct {
|
||||
ReplicationProxyRequest string
|
||||
}
|
||||
|
||||
// PutObjectTagging replaces or creates object tag(s) and can target
|
||||
// a specific object version in a versioned bucket.
|
||||
func (c *Client) PutObjectTagging(ctx context.Context, bucketName, objectName string, otags *tags.Tags, opts PutObjectTaggingOptions) error {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Get resources properly escaped and lined up before
|
||||
// using them in http request.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("tagging", "")
|
||||
|
||||
if opts.VersionID != "" {
|
||||
urlValues.Set("versionId", opts.VersionID)
|
||||
}
|
||||
headers := make(http.Header, 0)
|
||||
if opts.Internal.ReplicationProxyRequest != "" {
|
||||
headers.Set(minIOBucketReplicationProxyRequest, opts.Internal.ReplicationProxyRequest)
|
||||
}
|
||||
reqBytes, err := xml.Marshal(otags)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
reqMetadata := requestMetadata{
|
||||
bucketName: bucketName,
|
||||
objectName: objectName,
|
||||
queryValues: urlValues,
|
||||
contentBody: bytes.NewReader(reqBytes),
|
||||
contentLength: int64(len(reqBytes)),
|
||||
contentMD5Base64: sumMD5Base64(reqBytes),
|
||||
customHeader: headers,
|
||||
}
|
||||
|
||||
// Execute PUT to set a object tagging.
|
||||
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if resp != nil {
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return httpRespToErrorResponse(resp, bucketName, objectName)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetObjectTaggingOptions holds the object version ID
|
||||
// to fetch the tagging key/value pairs
|
||||
type GetObjectTaggingOptions struct {
|
||||
VersionID string
|
||||
Internal AdvancedObjectTaggingOptions
|
||||
}
|
||||
|
||||
// GetObjectTagging fetches object tag(s) with options to target
|
||||
// a specific object version in a versioned bucket.
|
||||
func (c *Client) GetObjectTagging(ctx context.Context, bucketName, objectName string, opts GetObjectTaggingOptions) (*tags.Tags, error) {
|
||||
// Get resources properly escaped and lined up before
|
||||
// using them in http request.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("tagging", "")
|
||||
|
||||
if opts.VersionID != "" {
|
||||
urlValues.Set("versionId", opts.VersionID)
|
||||
}
|
||||
headers := make(http.Header, 0)
|
||||
if opts.Internal.ReplicationProxyRequest != "" {
|
||||
headers.Set(minIOBucketReplicationProxyRequest, opts.Internal.ReplicationProxyRequest)
|
||||
}
|
||||
// Execute GET on object to get object tag(s)
|
||||
resp, err := c.executeMethod(ctx, http.MethodGet, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
objectName: objectName,
|
||||
queryValues: urlValues,
|
||||
customHeader: headers,
|
||||
})
|
||||
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if resp != nil {
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return nil, httpRespToErrorResponse(resp, bucketName, objectName)
|
||||
}
|
||||
}
|
||||
|
||||
return tags.ParseObjectXML(resp.Body)
|
||||
}
|
||||
|
||||
// RemoveObjectTaggingOptions holds the version id of the object to remove
|
||||
type RemoveObjectTaggingOptions struct {
|
||||
VersionID string
|
||||
Internal AdvancedObjectTaggingOptions
|
||||
}
|
||||
|
||||
// RemoveObjectTagging removes object tag(s) with options to control a specific object
|
||||
// version in a versioned bucket
|
||||
func (c *Client) RemoveObjectTagging(ctx context.Context, bucketName, objectName string, opts RemoveObjectTaggingOptions) error {
|
||||
// Get resources properly escaped and lined up before
|
||||
// using them in http request.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("tagging", "")
|
||||
|
||||
if opts.VersionID != "" {
|
||||
urlValues.Set("versionId", opts.VersionID)
|
||||
}
|
||||
headers := make(http.Header, 0)
|
||||
if opts.Internal.ReplicationProxyRequest != "" {
|
||||
headers.Set(minIOBucketReplicationProxyRequest, opts.Internal.ReplicationProxyRequest)
|
||||
}
|
||||
// Execute DELETE on object to remove object tag(s)
|
||||
resp, err := c.executeMethod(ctx, http.MethodDelete, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
objectName: objectName,
|
||||
queryValues: urlValues,
|
||||
customHeader: headers,
|
||||
})
|
||||
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if resp != nil {
|
||||
// S3 returns "204 No content" after Object tag deletion.
|
||||
if resp.StatusCode != http.StatusNoContent {
|
||||
return httpRespToErrorResponse(resp, bucketName, objectName)
|
||||
}
|
||||
}
|
||||
return err
|
||||
}
|
||||
+228
@@ -0,0 +1,228 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2015-2017 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"time"
|
||||
|
||||
"github.com/minio/minio-go/v7/pkg/s3utils"
|
||||
"github.com/minio/minio-go/v7/pkg/signer"
|
||||
)
|
||||
|
||||
// presignURL - Returns a presigned URL for an input 'method'.
|
||||
// Expires maximum is 7days - ie. 604800 and minimum is 1.
|
||||
func (c *Client) presignURL(ctx context.Context, method, bucketName, objectName string, expires time.Duration, reqParams url.Values, extraHeaders http.Header) (u *url.URL, err error) {
|
||||
// Input validation.
|
||||
if method == "" {
|
||||
return nil, errInvalidArgument("method cannot be empty.")
|
||||
}
|
||||
if err = s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err = isValidExpiry(expires); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Convert expires into seconds.
|
||||
expireSeconds := int64(expires / time.Second)
|
||||
reqMetadata := requestMetadata{
|
||||
presignURL: true,
|
||||
bucketName: bucketName,
|
||||
objectName: objectName,
|
||||
expires: expireSeconds,
|
||||
queryValues: reqParams,
|
||||
extraPresignHeader: extraHeaders,
|
||||
}
|
||||
|
||||
// Instantiate a new request.
|
||||
// Since expires is set newRequest will presign the request.
|
||||
var req *http.Request
|
||||
if req, err = c.newRequest(ctx, method, reqMetadata); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return req.URL, nil
|
||||
}
|
||||
|
||||
// PresignedGetObject - Returns a presigned URL to access an object
|
||||
// data without credentials. URL can have a maximum expiry of
|
||||
// upto 7days or a minimum of 1sec. Additionally you can override
|
||||
// a set of response headers using the query parameters.
|
||||
func (c *Client) PresignedGetObject(ctx context.Context, bucketName, objectName string, expires time.Duration, reqParams url.Values) (u *url.URL, err error) {
|
||||
if err = s3utils.CheckValidObjectName(objectName); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return c.presignURL(ctx, http.MethodGet, bucketName, objectName, expires, reqParams, nil)
|
||||
}
|
||||
|
||||
// PresignedHeadObject - Returns a presigned URL to access
|
||||
// object metadata without credentials. URL can have a maximum expiry
|
||||
// of upto 7days or a minimum of 1sec. Additionally you can override
|
||||
// a set of response headers using the query parameters.
|
||||
func (c *Client) PresignedHeadObject(ctx context.Context, bucketName, objectName string, expires time.Duration, reqParams url.Values) (u *url.URL, err error) {
|
||||
if err = s3utils.CheckValidObjectName(objectName); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return c.presignURL(ctx, http.MethodHead, bucketName, objectName, expires, reqParams, nil)
|
||||
}
|
||||
|
||||
// PresignedPutObject - Returns a presigned URL to upload an object
|
||||
// without credentials. URL can have a maximum expiry of upto 7days
|
||||
// or a minimum of 1sec.
|
||||
func (c *Client) PresignedPutObject(ctx context.Context, bucketName, objectName string, expires time.Duration) (u *url.URL, err error) {
|
||||
if err = s3utils.CheckValidObjectName(objectName); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return c.presignURL(ctx, http.MethodPut, bucketName, objectName, expires, nil, nil)
|
||||
}
|
||||
|
||||
// PresignHeader - similar to Presign() but allows including HTTP headers that
|
||||
// will be used to build the signature. The request using the resulting URL will
|
||||
// need to have the exact same headers to be added for signature validation to
|
||||
// pass.
|
||||
//
|
||||
// FIXME: The extra header parameter should be included in Presign() in the next
|
||||
// major version bump, and this function should then be deprecated.
|
||||
func (c *Client) PresignHeader(ctx context.Context, method, bucketName, objectName string, expires time.Duration, reqParams url.Values, extraHeaders http.Header) (u *url.URL, err error) {
|
||||
return c.presignURL(ctx, method, bucketName, objectName, expires, reqParams, extraHeaders)
|
||||
}
|
||||
|
||||
// Presign - returns a presigned URL for any http method of your choice along
|
||||
// with custom request params and extra signed headers. URL can have a maximum
|
||||
// expiry of upto 7days or a minimum of 1sec.
|
||||
func (c *Client) Presign(ctx context.Context, method, bucketName, objectName string, expires time.Duration, reqParams url.Values) (u *url.URL, err error) {
|
||||
return c.presignURL(ctx, method, bucketName, objectName, expires, reqParams, nil)
|
||||
}
|
||||
|
||||
// PresignedPostPolicy - Returns POST urlString, form data to upload an object.
|
||||
func (c *Client) PresignedPostPolicy(ctx context.Context, p *PostPolicy) (u *url.URL, formData map[string]string, err error) {
|
||||
// Validate input arguments.
|
||||
if p.expiration.IsZero() {
|
||||
return nil, nil, errors.New("Expiration time must be specified")
|
||||
}
|
||||
if _, ok := p.formData["key"]; !ok {
|
||||
return nil, nil, errors.New("object key must be specified")
|
||||
}
|
||||
if _, ok := p.formData["bucket"]; !ok {
|
||||
return nil, nil, errors.New("bucket name must be specified")
|
||||
}
|
||||
|
||||
bucketName := p.formData["bucket"]
|
||||
// Fetch the bucket location.
|
||||
location, err := c.getBucketLocation(ctx, bucketName)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
|
||||
isVirtualHost := c.isVirtualHostStyleRequest(*c.endpointURL, bucketName)
|
||||
|
||||
u, err = c.makeTargetURL(bucketName, "", location, isVirtualHost, nil)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
|
||||
// Get credentials from the configured credentials provider.
|
||||
credValues, err := c.credsProvider.GetWithContext(c.CredContext())
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
|
||||
var (
|
||||
signerType = credValues.SignerType
|
||||
sessionToken = credValues.SessionToken
|
||||
accessKeyID = credValues.AccessKeyID
|
||||
secretAccessKey = credValues.SecretAccessKey
|
||||
)
|
||||
|
||||
if signerType.IsAnonymous() {
|
||||
return nil, nil, errInvalidArgument("Presigned operations are not supported for anonymous credentials")
|
||||
}
|
||||
|
||||
// Keep time.
|
||||
t := time.Now().UTC()
|
||||
// For signature version '2' handle here.
|
||||
if signerType.IsV2() {
|
||||
policyBase64 := p.base64()
|
||||
p.formData["policy"] = policyBase64
|
||||
// For Google endpoint set this value to be 'GoogleAccessId'.
|
||||
if s3utils.IsGoogleEndpoint(*c.endpointURL) {
|
||||
p.formData["GoogleAccessId"] = accessKeyID
|
||||
} else {
|
||||
// For all other endpoints set this value to be 'AWSAccessKeyId'.
|
||||
p.formData["AWSAccessKeyId"] = accessKeyID
|
||||
}
|
||||
// Sign the policy.
|
||||
p.formData["signature"] = signer.PostPresignSignatureV2(policyBase64, secretAccessKey)
|
||||
return u, p.formData, nil
|
||||
}
|
||||
|
||||
// Add date policy.
|
||||
if err = p.addNewPolicy(policyCondition{
|
||||
matchType: "eq",
|
||||
condition: "$x-amz-date",
|
||||
value: t.Format(iso8601DateFormat),
|
||||
}); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
|
||||
// Add algorithm policy.
|
||||
if err = p.addNewPolicy(policyCondition{
|
||||
matchType: "eq",
|
||||
condition: "$x-amz-algorithm",
|
||||
value: signV4Algorithm,
|
||||
}); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
|
||||
// Add a credential policy.
|
||||
credential := signer.GetCredential(accessKeyID, location, t, signer.ServiceTypeS3)
|
||||
if err = p.addNewPolicy(policyCondition{
|
||||
matchType: "eq",
|
||||
condition: "$x-amz-credential",
|
||||
value: credential,
|
||||
}); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
|
||||
if sessionToken != "" {
|
||||
if err = p.addNewPolicy(policyCondition{
|
||||
matchType: "eq",
|
||||
condition: "$x-amz-security-token",
|
||||
value: sessionToken,
|
||||
}); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
}
|
||||
|
||||
// Get base64 encoded policy.
|
||||
policyBase64 := p.base64()
|
||||
|
||||
// Fill in the form data.
|
||||
p.formData["policy"] = policyBase64
|
||||
p.formData["x-amz-algorithm"] = signV4Algorithm
|
||||
p.formData["x-amz-credential"] = credential
|
||||
p.formData["x-amz-date"] = t.Format(iso8601DateFormat)
|
||||
if sessionToken != "" {
|
||||
p.formData["x-amz-security-token"] = sessionToken
|
||||
}
|
||||
p.formData["x-amz-signature"] = signer.PostPresignSignatureV4(policyBase64, t, secretAccessKey, location)
|
||||
return u, p.formData, nil
|
||||
}
|
||||
+78
@@ -0,0 +1,78 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2015-2024 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"io"
|
||||
"net/http"
|
||||
|
||||
"github.com/goccy/go-json"
|
||||
"github.com/minio/minio-go/v7/pkg/s3utils"
|
||||
)
|
||||
|
||||
// PromptObject performs language model inference with the prompt and referenced object as context.
|
||||
// Inference is performed using a Lambda handler that can process the prompt and object.
|
||||
// Currently, this functionality is limited to certain MinIO servers.
|
||||
func (c *Client) PromptObject(ctx context.Context, bucketName, objectName, prompt string, opts PromptObjectOptions) (io.ReadCloser, error) {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return nil, ErrorResponse{
|
||||
StatusCode: http.StatusBadRequest,
|
||||
Code: "InvalidBucketName",
|
||||
Message: err.Error(),
|
||||
}
|
||||
}
|
||||
if err := s3utils.CheckValidObjectName(objectName); err != nil {
|
||||
return nil, ErrorResponse{
|
||||
StatusCode: http.StatusBadRequest,
|
||||
Code: "XMinioInvalidObjectName",
|
||||
Message: err.Error(),
|
||||
}
|
||||
}
|
||||
|
||||
opts.AddLambdaArnToReqParams(opts.LambdaArn)
|
||||
opts.SetHeader("Content-Type", "application/json")
|
||||
opts.AddPromptArg("prompt", prompt)
|
||||
promptReqBytes, err := json.Marshal(opts.PromptArgs)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Execute POST on bucket/object.
|
||||
resp, err := c.executeMethod(ctx, http.MethodPost, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
objectName: objectName,
|
||||
queryValues: opts.toQueryValues(),
|
||||
customHeader: opts.Header(),
|
||||
contentSHA256Hex: sum256Hex(promptReqBytes),
|
||||
contentBody: bytes.NewReader(promptReqBytes),
|
||||
contentLength: int64(len(promptReqBytes)),
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
defer closeResponse(resp)
|
||||
return nil, httpRespToErrorResponse(resp, bucketName, objectName)
|
||||
}
|
||||
|
||||
return resp.Body, nil
|
||||
}
|
||||
+84
@@ -0,0 +1,84 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2015-2024 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/url"
|
||||
)
|
||||
|
||||
// PromptObjectOptions provides options to PromptObject call.
|
||||
// LambdaArn is the ARN of the Prompt Lambda to be invoked.
|
||||
// PromptArgs is a map of key-value pairs to be passed to the inference action on the Prompt Lambda.
|
||||
// "prompt" is a reserved key and should not be used as a key in PromptArgs.
|
||||
type PromptObjectOptions struct {
|
||||
LambdaArn string
|
||||
PromptArgs map[string]any
|
||||
headers map[string]string
|
||||
reqParams url.Values
|
||||
}
|
||||
|
||||
// Header returns the http.Header representation of the POST options.
|
||||
func (o PromptObjectOptions) Header() http.Header {
|
||||
headers := make(http.Header, len(o.headers))
|
||||
for k, v := range o.headers {
|
||||
headers.Set(k, v)
|
||||
}
|
||||
return headers
|
||||
}
|
||||
|
||||
// AddPromptArg Add a key value pair to the prompt arguments where the key is a string and
|
||||
// the value is a JSON serializable.
|
||||
func (o *PromptObjectOptions) AddPromptArg(key string, value any) {
|
||||
if o.PromptArgs == nil {
|
||||
o.PromptArgs = make(map[string]any)
|
||||
}
|
||||
o.PromptArgs[key] = value
|
||||
}
|
||||
|
||||
// AddLambdaArnToReqParams adds the lambdaArn to the request query string parameters.
|
||||
func (o *PromptObjectOptions) AddLambdaArnToReqParams(lambdaArn string) {
|
||||
if o.reqParams == nil {
|
||||
o.reqParams = make(url.Values)
|
||||
}
|
||||
o.reqParams.Add("lambdaArn", lambdaArn)
|
||||
}
|
||||
|
||||
// SetHeader adds a key value pair to the options. The
|
||||
// key-value pair will be part of the HTTP POST request
|
||||
// headers.
|
||||
func (o *PromptObjectOptions) SetHeader(key, value string) {
|
||||
if o.headers == nil {
|
||||
o.headers = make(map[string]string)
|
||||
}
|
||||
o.headers[http.CanonicalHeaderKey(key)] = value
|
||||
}
|
||||
|
||||
// toQueryValues - Convert the reqParams in Options to query string parameters.
|
||||
func (o *PromptObjectOptions) toQueryValues() url.Values {
|
||||
urlValues := make(url.Values)
|
||||
if o.reqParams != nil {
|
||||
for key, values := range o.reqParams {
|
||||
for _, value := range values {
|
||||
urlValues.Add(key, value)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return urlValues
|
||||
}
|
||||
+123
@@ -0,0 +1,123 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2015-2020 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/xml"
|
||||
"net/http"
|
||||
|
||||
"github.com/minio/minio-go/v7/pkg/s3utils"
|
||||
)
|
||||
|
||||
// Bucket operations
|
||||
func (c *Client) makeBucket(ctx context.Context, bucketName string, opts MakeBucketOptions) (err error) {
|
||||
// Validate the input arguments.
|
||||
if err := s3utils.CheckValidBucketNameStrict(bucketName); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
err = c.doMakeBucket(ctx, bucketName, opts.Region, opts.ObjectLocking)
|
||||
if err != nil && (opts.Region == "" || opts.Region == "us-east-1") {
|
||||
if resp, ok := err.(ErrorResponse); ok && resp.Code == "AuthorizationHeaderMalformed" && resp.Region != "" {
|
||||
err = c.doMakeBucket(ctx, bucketName, resp.Region, opts.ObjectLocking)
|
||||
}
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
func (c *Client) doMakeBucket(ctx context.Context, bucketName, location string, objectLockEnabled bool) (err error) {
|
||||
defer func() {
|
||||
// Save the location into cache on a successful makeBucket response.
|
||||
if err == nil {
|
||||
c.bucketLocCache.Set(bucketName, location)
|
||||
}
|
||||
}()
|
||||
|
||||
// If location is empty, treat is a default region 'us-east-1'.
|
||||
if location == "" {
|
||||
location = "us-east-1"
|
||||
// For custom region clients, default
|
||||
// to custom region instead not 'us-east-1'.
|
||||
if c.region != "" {
|
||||
location = c.region
|
||||
}
|
||||
}
|
||||
// PUT bucket request metadata.
|
||||
reqMetadata := requestMetadata{
|
||||
bucketName: bucketName,
|
||||
bucketLocation: location,
|
||||
}
|
||||
|
||||
if objectLockEnabled {
|
||||
headers := make(http.Header)
|
||||
headers.Add("x-amz-bucket-object-lock-enabled", "true")
|
||||
reqMetadata.customHeader = headers
|
||||
}
|
||||
|
||||
// If location is not 'us-east-1' create bucket location config.
|
||||
if location != "us-east-1" && location != "" {
|
||||
createBucketConfig := createBucketConfiguration{}
|
||||
createBucketConfig.Location = location
|
||||
var createBucketConfigBytes []byte
|
||||
createBucketConfigBytes, err = xml.Marshal(createBucketConfig)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
reqMetadata.contentMD5Base64 = sumMD5Base64(createBucketConfigBytes)
|
||||
reqMetadata.contentSHA256Hex = sum256Hex(createBucketConfigBytes)
|
||||
reqMetadata.contentBody = bytes.NewReader(createBucketConfigBytes)
|
||||
reqMetadata.contentLength = int64(len(createBucketConfigBytes))
|
||||
}
|
||||
|
||||
// Execute PUT to create a new bucket.
|
||||
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if resp != nil {
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
}
|
||||
|
||||
// Success.
|
||||
return nil
|
||||
}
|
||||
|
||||
// MakeBucketOptions holds all options to tweak bucket creation
|
||||
type MakeBucketOptions struct {
|
||||
// Bucket location
|
||||
Region string
|
||||
// Enable object locking
|
||||
ObjectLocking bool
|
||||
}
|
||||
|
||||
// MakeBucket creates a new bucket with bucketName with a context to control cancellations and timeouts.
|
||||
//
|
||||
// Location is an optional argument, by default all buckets are
|
||||
// created in US Standard Region.
|
||||
//
|
||||
// For Amazon S3 for more supported regions - http://docs.aws.amazon.com/general/latest/gr/rande.html
|
||||
// For Google Cloud Storage for more supported regions - https://cloud.google.com/storage/docs/bucket-locations
|
||||
func (c *Client) MakeBucket(ctx context.Context, bucketName string, opts MakeBucketOptions) (err error) {
|
||||
return c.makeBucket(ctx, bucketName, opts)
|
||||
}
|
||||
+149
@@ -0,0 +1,149 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2015-2017 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"math"
|
||||
"os"
|
||||
|
||||
"github.com/minio/minio-go/v7/pkg/s3utils"
|
||||
)
|
||||
|
||||
const nullVersionID = "null"
|
||||
|
||||
// Verify if reader is *minio.Object
|
||||
func isObject(reader io.Reader) (ok bool) {
|
||||
_, ok = reader.(*Object)
|
||||
return
|
||||
}
|
||||
|
||||
// Verify if reader is a generic ReaderAt
|
||||
func isReadAt(reader io.Reader) (ok bool) {
|
||||
var v *os.File
|
||||
v, ok = reader.(*os.File)
|
||||
if ok {
|
||||
// Stdin, Stdout and Stderr all have *os.File type
|
||||
// which happen to also be io.ReaderAt compatible
|
||||
// we need to add special conditions for them to
|
||||
// be ignored by this function.
|
||||
for _, f := range []string{
|
||||
"/dev/stdin",
|
||||
"/dev/stdout",
|
||||
"/dev/stderr",
|
||||
} {
|
||||
if f == v.Name() {
|
||||
ok = false
|
||||
break
|
||||
}
|
||||
}
|
||||
} else {
|
||||
_, ok = reader.(io.ReaderAt)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// OptimalPartInfo - calculate the optimal part info for a given
|
||||
// object size.
|
||||
//
|
||||
// NOTE: Assumption here is that for any object to be uploaded to any S3 compatible
|
||||
// object storage it will have the following parameters as constants.
|
||||
//
|
||||
// maxPartsCount - 10000
|
||||
// minPartSize - 16MiB
|
||||
// maxMultipartPutObjectSize - 5TiB
|
||||
func OptimalPartInfo(objectSize int64, configuredPartSize uint64) (totalPartsCount int, partSize, lastPartSize int64, err error) {
|
||||
// object size is '-1' set it to 5TiB.
|
||||
var unknownSize bool
|
||||
if objectSize == -1 {
|
||||
unknownSize = true
|
||||
objectSize = maxMultipartPutObjectSize
|
||||
}
|
||||
|
||||
// object size is larger than supported maximum.
|
||||
if objectSize > maxMultipartPutObjectSize {
|
||||
err = errEntityTooLarge(objectSize, maxMultipartPutObjectSize, "", "")
|
||||
return
|
||||
}
|
||||
|
||||
var partSizeFlt float64
|
||||
if configuredPartSize > 0 {
|
||||
if int64(configuredPartSize) > objectSize {
|
||||
err = errEntityTooLarge(int64(configuredPartSize), objectSize, "", "")
|
||||
return
|
||||
}
|
||||
|
||||
if !unknownSize {
|
||||
if objectSize > (int64(configuredPartSize) * maxPartsCount) {
|
||||
err = errInvalidArgument("Part size * max_parts(10000) is lesser than input objectSize.")
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
if configuredPartSize < absMinPartSize {
|
||||
err = errInvalidArgument("Input part size is smaller than allowed minimum of 5MiB.")
|
||||
return
|
||||
}
|
||||
|
||||
if configuredPartSize > maxPartSize {
|
||||
err = errInvalidArgument("Input part size is bigger than allowed maximum of 5GiB.")
|
||||
return
|
||||
}
|
||||
|
||||
partSizeFlt = float64(configuredPartSize)
|
||||
if unknownSize {
|
||||
// If input has unknown size and part size is configured
|
||||
// keep it to maximum allowed as per 10000 parts.
|
||||
objectSize = int64(configuredPartSize) * maxPartsCount
|
||||
}
|
||||
} else {
|
||||
configuredPartSize = minPartSize
|
||||
// Use floats for part size for all calculations to avoid
|
||||
// overflows during float64 to int64 conversions.
|
||||
partSizeFlt = float64(objectSize / maxPartsCount)
|
||||
partSizeFlt = math.Ceil(partSizeFlt/float64(configuredPartSize)) * float64(configuredPartSize)
|
||||
}
|
||||
|
||||
// Total parts count.
|
||||
totalPartsCount = int(math.Ceil(float64(objectSize) / partSizeFlt))
|
||||
// Part size.
|
||||
partSize = int64(partSizeFlt)
|
||||
// Last part size.
|
||||
lastPartSize = objectSize - int64(totalPartsCount-1)*partSize
|
||||
return totalPartsCount, partSize, lastPartSize, nil
|
||||
}
|
||||
|
||||
// getUploadID - fetch upload id if already present for an object name
|
||||
// or initiate a new request to fetch a new upload id.
|
||||
func (c *Client) newUploadID(ctx context.Context, bucketName, objectName string, opts PutObjectOptions) (uploadID string, err error) {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err := s3utils.CheckValidObjectName(objectName); err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
// Initiate multipart upload for an object.
|
||||
initMultipartUploadResult, err := c.initiateMultipartUpload(ctx, bucketName, objectName, opts)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return initMultipartUploadResult.UploadID, nil
|
||||
}
|
||||
+167
@@ -0,0 +1,167 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2023 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"mime/multipart"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/minio/minio-go/v7/pkg/encrypt"
|
||||
)
|
||||
|
||||
// PutObjectFanOutEntry is per object entry fan-out metadata
|
||||
type PutObjectFanOutEntry struct {
|
||||
Key string `json:"key"`
|
||||
UserMetadata map[string]string `json:"metadata,omitempty"`
|
||||
UserTags map[string]string `json:"tags,omitempty"`
|
||||
ContentType string `json:"contentType,omitempty"`
|
||||
ContentEncoding string `json:"contentEncoding,omitempty"`
|
||||
ContentDisposition string `json:"contentDisposition,omitempty"`
|
||||
ContentLanguage string `json:"contentLanguage,omitempty"`
|
||||
CacheControl string `json:"cacheControl,omitempty"`
|
||||
Retention RetentionMode `json:"retention,omitempty"`
|
||||
RetainUntilDate *time.Time `json:"retainUntil,omitempty"`
|
||||
}
|
||||
|
||||
// PutObjectFanOutRequest this is the request structure sent
|
||||
// to the server to fan-out the stream to multiple objects.
|
||||
type PutObjectFanOutRequest struct {
|
||||
Entries []PutObjectFanOutEntry
|
||||
Checksum Checksum
|
||||
SSE encrypt.ServerSide
|
||||
}
|
||||
|
||||
// PutObjectFanOutResponse this is the response structure sent
|
||||
// by the server upon success or failure for each object
|
||||
// fan-out keys. Additionally, this response carries ETag,
|
||||
// VersionID and LastModified for each object fan-out.
|
||||
type PutObjectFanOutResponse struct {
|
||||
Key string `json:"key"`
|
||||
ETag string `json:"etag,omitempty"`
|
||||
VersionID string `json:"versionId,omitempty"`
|
||||
LastModified *time.Time `json:"lastModified,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// PutObjectFanOut - is a variant of PutObject instead of writing a single object from a single
|
||||
// stream multiple objects are written, defined via a list of PutObjectFanOutRequests. Each entry
|
||||
// in PutObjectFanOutRequest carries an object keyname and its relevant metadata if any. `Key` is
|
||||
// mandatory, rest of the other options in PutObjectFanOutRequest are optional.
|
||||
func (c *Client) PutObjectFanOut(ctx context.Context, bucket string, fanOutData io.Reader, fanOutReq PutObjectFanOutRequest) ([]PutObjectFanOutResponse, error) {
|
||||
if len(fanOutReq.Entries) == 0 {
|
||||
return nil, errInvalidArgument("fan out requests cannot be empty")
|
||||
}
|
||||
|
||||
policy := NewPostPolicy()
|
||||
policy.SetBucket(bucket)
|
||||
policy.SetKey(strconv.FormatInt(time.Now().UnixNano(), 16))
|
||||
|
||||
// Expires in 15 minutes.
|
||||
policy.SetExpires(time.Now().UTC().Add(15 * time.Minute))
|
||||
|
||||
// Set encryption headers if any.
|
||||
policy.SetEncryption(fanOutReq.SSE)
|
||||
|
||||
// Set checksum headers if any.
|
||||
err := policy.SetChecksum(fanOutReq.Checksum)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
url, formData, err := c.PresignedPostPolicy(ctx, policy)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
r, w := io.Pipe()
|
||||
|
||||
req, err := http.NewRequest(http.MethodPost, url.String(), r)
|
||||
if err != nil {
|
||||
w.Close()
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var b strings.Builder
|
||||
enc := json.NewEncoder(&b)
|
||||
for _, req := range fanOutReq.Entries {
|
||||
if req.Key == "" {
|
||||
w.Close()
|
||||
return nil, errors.New("PutObjectFanOutRequest.Key is mandatory and cannot be empty")
|
||||
}
|
||||
if err = enc.Encode(&req); err != nil {
|
||||
w.Close()
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
mwriter := multipart.NewWriter(w)
|
||||
req.Header.Add("Content-Type", mwriter.FormDataContentType())
|
||||
|
||||
go func() {
|
||||
defer w.Close()
|
||||
defer mwriter.Close()
|
||||
|
||||
for k, v := range formData {
|
||||
if err := mwriter.WriteField(k, v); err != nil {
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
if err := mwriter.WriteField("x-minio-fanout-list", b.String()); err != nil {
|
||||
return
|
||||
}
|
||||
|
||||
mw, err := mwriter.CreateFormFile("file", "fanout-content")
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
|
||||
if _, err = io.Copy(mw, fanOutData); err != nil {
|
||||
return
|
||||
}
|
||||
}()
|
||||
|
||||
resp, err := c.do(req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer closeResponse(resp)
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return nil, httpRespToErrorResponse(resp, bucket, "fanout-content")
|
||||
}
|
||||
|
||||
dec := json.NewDecoder(resp.Body)
|
||||
fanOutResp := make([]PutObjectFanOutResponse, 0, len(fanOutReq.Entries))
|
||||
for dec.More() {
|
||||
var m PutObjectFanOutResponse
|
||||
if err = dec.Decode(&m); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
fanOutResp = append(fanOutResp, m)
|
||||
}
|
||||
|
||||
return fanOutResp, nil
|
||||
}
|
||||
Generated
Vendored
+64
@@ -0,0 +1,64 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2017 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"context"
|
||||
"mime"
|
||||
"os"
|
||||
"path/filepath"
|
||||
|
||||
"github.com/minio/minio-go/v7/pkg/s3utils"
|
||||
)
|
||||
|
||||
// FPutObject - Create an object in a bucket, with contents from file at filePath. Allows request cancellation.
|
||||
func (c *Client) FPutObject(ctx context.Context, bucketName, objectName, filePath string, opts PutObjectOptions) (info UploadInfo, err error) {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
if err := s3utils.CheckValidObjectName(objectName); err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
|
||||
// Open the referenced file.
|
||||
fileReader, err := os.Open(filePath)
|
||||
// If any error fail quickly here.
|
||||
if err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
defer fileReader.Close()
|
||||
|
||||
// Save the file stat.
|
||||
fileStat, err := fileReader.Stat()
|
||||
if err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
|
||||
// Save the file size.
|
||||
fileSize := fileStat.Size()
|
||||
|
||||
// Set contentType based on filepath extension if not given or default
|
||||
// value of "application/octet-stream" if the extension has no associated type.
|
||||
if opts.ContentType == "" {
|
||||
if opts.ContentType = mime.TypeByExtension(filepath.Ext(filePath)); opts.ContentType == "" {
|
||||
opts.ContentType = "application/octet-stream"
|
||||
}
|
||||
}
|
||||
return c.PutObject(ctx, bucketName, objectName, fileReader, fileSize, opts)
|
||||
}
|
||||
+461
@@ -0,0 +1,461 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2015-2017 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/hex"
|
||||
"encoding/xml"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/minio/minio-go/v7/pkg/encrypt"
|
||||
"github.com/minio/minio-go/v7/pkg/s3utils"
|
||||
)
|
||||
|
||||
func (c *Client) putObjectMultipart(ctx context.Context, bucketName, objectName string, reader io.Reader, size int64,
|
||||
opts PutObjectOptions,
|
||||
) (info UploadInfo, err error) {
|
||||
info, err = c.putObjectMultipartNoStream(ctx, bucketName, objectName, reader, opts)
|
||||
if err != nil {
|
||||
errResp := ToErrorResponse(err)
|
||||
// Verify if multipart functionality is not available, if not
|
||||
// fall back to single PutObject operation.
|
||||
if errResp.Code == "AccessDenied" && strings.Contains(errResp.Message, "Access Denied") {
|
||||
// Verify if size of reader is greater than '5GiB'.
|
||||
if size > maxSinglePutObjectSize {
|
||||
return UploadInfo{}, errEntityTooLarge(size, maxSinglePutObjectSize, bucketName, objectName)
|
||||
}
|
||||
// Fall back to uploading as single PutObject operation.
|
||||
return c.putObject(ctx, bucketName, objectName, reader, size, opts)
|
||||
}
|
||||
}
|
||||
return info, err
|
||||
}
|
||||
|
||||
func (c *Client) putObjectMultipartNoStream(ctx context.Context, bucketName, objectName string, reader io.Reader, opts PutObjectOptions) (info UploadInfo, err error) {
|
||||
// Input validation.
|
||||
if err = s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
if err = s3utils.CheckValidObjectName(objectName); err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
|
||||
// Total data read and written to server. should be equal to
|
||||
// 'size' at the end of the call.
|
||||
var totalUploadedSize int64
|
||||
|
||||
// Complete multipart upload.
|
||||
var complMultipartUpload completeMultipartUpload
|
||||
|
||||
// Calculate the optimal parts info for a given size.
|
||||
totalPartsCount, partSize, _, err := OptimalPartInfo(-1, opts.PartSize)
|
||||
if err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
|
||||
// Choose hash algorithms to be calculated by hashCopyN,
|
||||
// avoid sha256 with non-v4 signature request or
|
||||
// HTTPS connection.
|
||||
hashAlgos, hashSums := c.hashMaterials(opts.SendContentMd5, !opts.DisableContentSha256)
|
||||
if len(hashSums) == 0 {
|
||||
addAutoChecksumHeaders(&opts)
|
||||
}
|
||||
|
||||
// Initiate a new multipart upload.
|
||||
uploadID, err := c.newUploadID(ctx, bucketName, objectName, opts)
|
||||
if err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
delete(opts.UserMetadata, "X-Amz-Checksum-Algorithm")
|
||||
|
||||
defer func() {
|
||||
if err != nil {
|
||||
c.abortMultipartUpload(ctx, bucketName, objectName, uploadID)
|
||||
}
|
||||
}()
|
||||
|
||||
// Part number always starts with '1'.
|
||||
partNumber := 1
|
||||
|
||||
// Initialize parts uploaded map.
|
||||
partsInfo := make(map[int]ObjectPart)
|
||||
|
||||
// Create a buffer.
|
||||
buf := make([]byte, partSize)
|
||||
|
||||
// Create checksums
|
||||
// CRC32C is ~50% faster on AMD64 @ 30GB/s
|
||||
customHeader := make(http.Header)
|
||||
crc := opts.AutoChecksum.Hasher()
|
||||
for partNumber <= totalPartsCount {
|
||||
length, rErr := readFull(reader, buf)
|
||||
if rErr == io.EOF && partNumber > 1 {
|
||||
break
|
||||
}
|
||||
|
||||
if rErr != nil && rErr != io.ErrUnexpectedEOF && rErr != io.EOF {
|
||||
return UploadInfo{}, rErr
|
||||
}
|
||||
|
||||
// Calculates hash sums while copying partSize bytes into cw.
|
||||
for k, v := range hashAlgos {
|
||||
v.Write(buf[:length])
|
||||
hashSums[k] = v.Sum(nil)
|
||||
v.Close()
|
||||
}
|
||||
|
||||
// Update progress reader appropriately to the latest offset
|
||||
// as we read from the source.
|
||||
rd := newHook(bytes.NewReader(buf[:length]), opts.Progress)
|
||||
|
||||
// Checksums..
|
||||
var (
|
||||
md5Base64 string
|
||||
sha256Hex string
|
||||
)
|
||||
|
||||
if hashSums["md5"] != nil {
|
||||
md5Base64 = base64.StdEncoding.EncodeToString(hashSums["md5"])
|
||||
}
|
||||
if hashSums["sha256"] != nil {
|
||||
sha256Hex = hex.EncodeToString(hashSums["sha256"])
|
||||
}
|
||||
if len(hashSums) == 0 {
|
||||
crc.Reset()
|
||||
crc.Write(buf[:length])
|
||||
cSum := crc.Sum(nil)
|
||||
customHeader.Set(opts.AutoChecksum.Key(), base64.StdEncoding.EncodeToString(cSum))
|
||||
}
|
||||
|
||||
p := uploadPartParams{bucketName: bucketName, objectName: objectName, uploadID: uploadID, reader: rd, partNumber: partNumber, md5Base64: md5Base64, sha256Hex: sha256Hex, size: int64(length), sse: opts.ServerSideEncryption, streamSha256: !opts.DisableContentSha256, customHeader: customHeader}
|
||||
// Proceed to upload the part.
|
||||
objPart, uerr := c.uploadPart(ctx, p)
|
||||
if uerr != nil {
|
||||
return UploadInfo{}, uerr
|
||||
}
|
||||
|
||||
// Save successfully uploaded part metadata.
|
||||
partsInfo[partNumber] = objPart
|
||||
|
||||
// Save successfully uploaded size.
|
||||
totalUploadedSize += int64(length)
|
||||
|
||||
// Increment part number.
|
||||
partNumber++
|
||||
|
||||
// For unknown size, Read EOF we break away.
|
||||
// We do not have to upload till totalPartsCount.
|
||||
if rErr == io.EOF {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
// Loop over total uploaded parts to save them in
|
||||
// Parts array before completing the multipart request.
|
||||
allParts := make([]ObjectPart, 0, len(partsInfo))
|
||||
for i := 1; i < partNumber; i++ {
|
||||
part, ok := partsInfo[i]
|
||||
if !ok {
|
||||
return UploadInfo{}, errInvalidArgument(fmt.Sprintf("Missing part number %d", i))
|
||||
}
|
||||
allParts = append(allParts, part)
|
||||
complMultipartUpload.Parts = append(complMultipartUpload.Parts, CompletePart{
|
||||
ETag: part.ETag,
|
||||
PartNumber: part.PartNumber,
|
||||
ChecksumCRC32: part.ChecksumCRC32,
|
||||
ChecksumCRC32C: part.ChecksumCRC32C,
|
||||
ChecksumSHA1: part.ChecksumSHA1,
|
||||
ChecksumSHA256: part.ChecksumSHA256,
|
||||
ChecksumCRC64NVME: part.ChecksumCRC64NVME,
|
||||
})
|
||||
}
|
||||
|
||||
// Sort all completed parts.
|
||||
sort.Sort(completedParts(complMultipartUpload.Parts))
|
||||
opts = PutObjectOptions{
|
||||
ServerSideEncryption: opts.ServerSideEncryption,
|
||||
AutoChecksum: opts.AutoChecksum,
|
||||
}
|
||||
applyAutoChecksum(&opts, allParts)
|
||||
|
||||
uploadInfo, err := c.completeMultipartUpload(ctx, bucketName, objectName, uploadID, complMultipartUpload, opts)
|
||||
if err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
|
||||
uploadInfo.Size = totalUploadedSize
|
||||
return uploadInfo, nil
|
||||
}
|
||||
|
||||
// initiateMultipartUpload - Initiates a multipart upload and returns an upload ID.
|
||||
func (c *Client) initiateMultipartUpload(ctx context.Context, bucketName, objectName string, opts PutObjectOptions) (initiateMultipartUploadResult, error) {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return initiateMultipartUploadResult{}, err
|
||||
}
|
||||
if err := s3utils.CheckValidObjectName(objectName); err != nil {
|
||||
return initiateMultipartUploadResult{}, err
|
||||
}
|
||||
|
||||
// Initialize url queries.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("uploads", "")
|
||||
|
||||
if opts.Internal.SourceVersionID != "" {
|
||||
if opts.Internal.SourceVersionID != nullVersionID {
|
||||
if _, err := uuid.Parse(opts.Internal.SourceVersionID); err != nil {
|
||||
return initiateMultipartUploadResult{}, errInvalidArgument(err.Error())
|
||||
}
|
||||
}
|
||||
urlValues.Set("versionId", opts.Internal.SourceVersionID)
|
||||
}
|
||||
|
||||
// Set ContentType header.
|
||||
customHeader := opts.Header()
|
||||
|
||||
reqMetadata := requestMetadata{
|
||||
bucketName: bucketName,
|
||||
objectName: objectName,
|
||||
queryValues: urlValues,
|
||||
customHeader: customHeader,
|
||||
}
|
||||
|
||||
// Execute POST on an objectName to initiate multipart upload.
|
||||
resp, err := c.executeMethod(ctx, http.MethodPost, reqMetadata)
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return initiateMultipartUploadResult{}, err
|
||||
}
|
||||
if resp != nil {
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return initiateMultipartUploadResult{}, httpRespToErrorResponse(resp, bucketName, objectName)
|
||||
}
|
||||
}
|
||||
// Decode xml for new multipart upload.
|
||||
initiateMultipartUploadResult := initiateMultipartUploadResult{}
|
||||
err = xmlDecoder(resp.Body, &initiateMultipartUploadResult)
|
||||
if err != nil {
|
||||
return initiateMultipartUploadResult, err
|
||||
}
|
||||
return initiateMultipartUploadResult, nil
|
||||
}
|
||||
|
||||
type uploadPartParams struct {
|
||||
bucketName string
|
||||
objectName string
|
||||
uploadID string
|
||||
reader io.Reader
|
||||
partNumber int
|
||||
md5Base64 string
|
||||
sha256Hex string
|
||||
size int64
|
||||
sse encrypt.ServerSide
|
||||
streamSha256 bool
|
||||
customHeader http.Header
|
||||
trailer http.Header
|
||||
}
|
||||
|
||||
// uploadPart - Uploads a part in a multipart upload.
|
||||
func (c *Client) uploadPart(ctx context.Context, p uploadPartParams) (ObjectPart, error) {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(p.bucketName); err != nil {
|
||||
return ObjectPart{}, err
|
||||
}
|
||||
if err := s3utils.CheckValidObjectName(p.objectName); err != nil {
|
||||
return ObjectPart{}, err
|
||||
}
|
||||
if p.size > maxPartSize {
|
||||
return ObjectPart{}, errEntityTooLarge(p.size, maxPartSize, p.bucketName, p.objectName)
|
||||
}
|
||||
if p.size <= -1 {
|
||||
return ObjectPart{}, errEntityTooSmall(p.size, p.bucketName, p.objectName)
|
||||
}
|
||||
if p.partNumber <= 0 {
|
||||
return ObjectPart{}, errInvalidArgument("Part number cannot be negative or equal to zero.")
|
||||
}
|
||||
if p.uploadID == "" {
|
||||
return ObjectPart{}, errInvalidArgument("UploadID cannot be empty.")
|
||||
}
|
||||
|
||||
// Get resources properly escaped and lined up before using them in http request.
|
||||
urlValues := make(url.Values)
|
||||
// Set part number.
|
||||
urlValues.Set("partNumber", strconv.Itoa(p.partNumber))
|
||||
// Set upload id.
|
||||
urlValues.Set("uploadId", p.uploadID)
|
||||
|
||||
// Set encryption headers, if any.
|
||||
if p.customHeader == nil {
|
||||
p.customHeader = make(http.Header)
|
||||
}
|
||||
// https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html
|
||||
// Server-side encryption is supported by the S3 Multipart Upload actions.
|
||||
// Unless you are using a customer-provided encryption key, you don't need
|
||||
// to specify the encryption parameters in each UploadPart request.
|
||||
if p.sse != nil && p.sse.Type() == encrypt.SSEC {
|
||||
p.sse.Marshal(p.customHeader)
|
||||
}
|
||||
|
||||
reqMetadata := requestMetadata{
|
||||
bucketName: p.bucketName,
|
||||
objectName: p.objectName,
|
||||
queryValues: urlValues,
|
||||
customHeader: p.customHeader,
|
||||
contentBody: p.reader,
|
||||
contentLength: p.size,
|
||||
contentMD5Base64: p.md5Base64,
|
||||
contentSHA256Hex: p.sha256Hex,
|
||||
streamSha256: p.streamSha256,
|
||||
trailer: p.trailer,
|
||||
}
|
||||
|
||||
// Execute PUT on each part.
|
||||
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return ObjectPart{}, err
|
||||
}
|
||||
if resp != nil {
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return ObjectPart{}, httpRespToErrorResponse(resp, p.bucketName, p.objectName)
|
||||
}
|
||||
}
|
||||
// Once successfully uploaded, return completed part.
|
||||
h := resp.Header
|
||||
objPart := ObjectPart{
|
||||
ChecksumCRC32: h.Get(ChecksumCRC32.Key()),
|
||||
ChecksumCRC32C: h.Get(ChecksumCRC32C.Key()),
|
||||
ChecksumSHA1: h.Get(ChecksumSHA1.Key()),
|
||||
ChecksumSHA256: h.Get(ChecksumSHA256.Key()),
|
||||
ChecksumCRC64NVME: h.Get(ChecksumCRC64NVME.Key()),
|
||||
}
|
||||
objPart.Size = p.size
|
||||
objPart.PartNumber = p.partNumber
|
||||
// Trim off the odd double quotes from ETag in the beginning and end.
|
||||
objPart.ETag = trimEtag(h.Get("ETag"))
|
||||
return objPart, nil
|
||||
}
|
||||
|
||||
// completeMultipartUpload - Completes a multipart upload by assembling previously uploaded parts.
|
||||
func (c *Client) completeMultipartUpload(ctx context.Context, bucketName, objectName, uploadID string,
|
||||
complete completeMultipartUpload, opts PutObjectOptions,
|
||||
) (UploadInfo, error) {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
if err := s3utils.CheckValidObjectName(objectName); err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
|
||||
// Initialize url queries.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("uploadId", uploadID)
|
||||
// Marshal complete multipart body.
|
||||
completeMultipartUploadBytes, err := xml.Marshal(complete)
|
||||
if err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
|
||||
headers := opts.Header()
|
||||
if s3utils.IsAmazonEndpoint(*c.endpointURL) {
|
||||
headers.Del(encrypt.SseKmsKeyID) // Remove X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id not supported in CompleteMultipartUpload
|
||||
headers.Del(encrypt.SseGenericHeader) // Remove X-Amz-Server-Side-Encryption not supported in CompleteMultipartUpload
|
||||
headers.Del(encrypt.SseEncryptionContext) // Remove X-Amz-Server-Side-Encryption-Context not supported in CompleteMultipartUpload
|
||||
}
|
||||
|
||||
// Instantiate all the complete multipart buffer.
|
||||
completeMultipartUploadBuffer := bytes.NewReader(completeMultipartUploadBytes)
|
||||
reqMetadata := requestMetadata{
|
||||
bucketName: bucketName,
|
||||
objectName: objectName,
|
||||
queryValues: urlValues,
|
||||
contentBody: completeMultipartUploadBuffer,
|
||||
contentLength: int64(len(completeMultipartUploadBytes)),
|
||||
contentSHA256Hex: sum256Hex(completeMultipartUploadBytes),
|
||||
customHeader: headers,
|
||||
}
|
||||
|
||||
// Execute POST to complete multipart upload for an objectName.
|
||||
resp, err := c.executeMethod(ctx, http.MethodPost, reqMetadata)
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
if resp != nil {
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return UploadInfo{}, httpRespToErrorResponse(resp, bucketName, objectName)
|
||||
}
|
||||
}
|
||||
|
||||
// Read resp.Body into a []bytes to parse for Error response inside the body
|
||||
var b []byte
|
||||
b, err = io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
// Decode completed multipart upload response on success.
|
||||
completeMultipartUploadResult := completeMultipartUploadResult{}
|
||||
err = xmlDecoder(bytes.NewReader(b), &completeMultipartUploadResult)
|
||||
if err != nil {
|
||||
// xml parsing failure due to presence an ill-formed xml fragment
|
||||
return UploadInfo{}, err
|
||||
} else if completeMultipartUploadResult.Bucket == "" {
|
||||
// xml's Decode method ignores well-formed xml that don't apply to the type of value supplied.
|
||||
// In this case, it would leave completeMultipartUploadResult with the corresponding zero-values
|
||||
// of the members.
|
||||
|
||||
// Decode completed multipart upload response on failure
|
||||
completeMultipartUploadErr := ErrorResponse{}
|
||||
err = xmlDecoder(bytes.NewReader(b), &completeMultipartUploadErr)
|
||||
if err != nil {
|
||||
// xml parsing failure due to presence an ill-formed xml fragment
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
return UploadInfo{}, completeMultipartUploadErr
|
||||
}
|
||||
|
||||
// extract lifecycle expiry date and rule ID
|
||||
expTime, ruleID := amzExpirationToExpiryDateRuleID(resp.Header.Get(amzExpiration))
|
||||
|
||||
return UploadInfo{
|
||||
Bucket: completeMultipartUploadResult.Bucket,
|
||||
Key: completeMultipartUploadResult.Key,
|
||||
ETag: trimEtag(completeMultipartUploadResult.ETag),
|
||||
VersionID: resp.Header.Get(amzVersionID),
|
||||
Location: completeMultipartUploadResult.Location,
|
||||
Expiration: expTime,
|
||||
ExpirationRuleID: ruleID,
|
||||
|
||||
ChecksumSHA256: completeMultipartUploadResult.ChecksumSHA256,
|
||||
ChecksumSHA1: completeMultipartUploadResult.ChecksumSHA1,
|
||||
ChecksumCRC32: completeMultipartUploadResult.ChecksumCRC32,
|
||||
ChecksumCRC32C: completeMultipartUploadResult.ChecksumCRC32C,
|
||||
ChecksumCRC64NVME: completeMultipartUploadResult.ChecksumCRC64NVME,
|
||||
}, nil
|
||||
}
|
||||
+810
@@ -0,0 +1,810 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2017 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/minio/minio-go/v7/pkg/s3utils"
|
||||
)
|
||||
|
||||
// putObjectMultipartStream - upload a large object using
|
||||
// multipart upload and streaming signature for signing payload.
|
||||
// Comprehensive put object operation involving multipart uploads.
|
||||
//
|
||||
// Following code handles these types of readers.
|
||||
//
|
||||
// - *minio.Object
|
||||
// - Any reader which has a method 'ReadAt()'
|
||||
func (c *Client) putObjectMultipartStream(ctx context.Context, bucketName, objectName string,
|
||||
reader io.Reader, size int64, opts PutObjectOptions,
|
||||
) (info UploadInfo, err error) {
|
||||
if opts.ConcurrentStreamParts && opts.NumThreads > 1 {
|
||||
info, err = c.putObjectMultipartStreamParallel(ctx, bucketName, objectName, reader, opts)
|
||||
} else if !isObject(reader) && isReadAt(reader) && !opts.SendContentMd5 {
|
||||
// Verify if the reader implements ReadAt and it is not a *minio.Object then we will use parallel uploader.
|
||||
info, err = c.putObjectMultipartStreamFromReadAt(ctx, bucketName, objectName, reader.(io.ReaderAt), size, opts)
|
||||
} else {
|
||||
info, err = c.putObjectMultipartStreamOptionalChecksum(ctx, bucketName, objectName, reader, size, opts)
|
||||
}
|
||||
if err != nil && s3utils.IsGoogleEndpoint(*c.endpointURL) {
|
||||
errResp := ToErrorResponse(err)
|
||||
// Verify if multipart functionality is not available, if not
|
||||
// fall back to single PutObject operation.
|
||||
if errResp.Code == "AccessDenied" && strings.Contains(errResp.Message, "Access Denied") {
|
||||
// Verify if size of reader is greater than '5GiB'.
|
||||
if size > maxSinglePutObjectSize {
|
||||
return UploadInfo{}, errEntityTooLarge(size, maxSinglePutObjectSize, bucketName, objectName)
|
||||
}
|
||||
// Fall back to uploading as single PutObject operation.
|
||||
return c.putObject(ctx, bucketName, objectName, reader, size, opts)
|
||||
}
|
||||
}
|
||||
return info, err
|
||||
}
|
||||
|
||||
// uploadedPartRes - the response received from a part upload.
|
||||
type uploadedPartRes struct {
|
||||
Error error // Any error encountered while uploading the part.
|
||||
PartNum int // Number of the part uploaded.
|
||||
Size int64 // Size of the part uploaded.
|
||||
Part ObjectPart
|
||||
}
|
||||
|
||||
type uploadPartReq struct {
|
||||
PartNum int // Number of the part uploaded.
|
||||
Part ObjectPart // Size of the part uploaded.
|
||||
}
|
||||
|
||||
// putObjectMultipartFromReadAt - Uploads files bigger than 128MiB.
|
||||
// Supports all readers which implements io.ReaderAt interface
|
||||
// (ReadAt method).
|
||||
//
|
||||
// NOTE: This function is meant to be used for all readers which
|
||||
// implement io.ReaderAt which allows us for resuming multipart
|
||||
// uploads but reading at an offset, which would avoid re-read the
|
||||
// data which was already uploaded. Internally this function uses
|
||||
// temporary files for staging all the data, these temporary files are
|
||||
// cleaned automatically when the caller i.e http client closes the
|
||||
// stream after uploading all the contents successfully.
|
||||
func (c *Client) putObjectMultipartStreamFromReadAt(ctx context.Context, bucketName, objectName string,
|
||||
reader io.ReaderAt, size int64, opts PutObjectOptions,
|
||||
) (info UploadInfo, err error) {
|
||||
// Input validation.
|
||||
if err = s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
if err = s3utils.CheckValidObjectName(objectName); err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
|
||||
// Calculate the optimal parts info for a given size.
|
||||
totalPartsCount, partSize, lastPartSize, err := OptimalPartInfo(size, opts.PartSize)
|
||||
if err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
if opts.Checksum.IsSet() {
|
||||
opts.AutoChecksum = opts.Checksum
|
||||
}
|
||||
withChecksum := c.trailingHeaderSupport
|
||||
if withChecksum {
|
||||
addAutoChecksumHeaders(&opts)
|
||||
}
|
||||
// Initiate a new multipart upload.
|
||||
uploadID, err := c.newUploadID(ctx, bucketName, objectName, opts)
|
||||
if err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
delete(opts.UserMetadata, "X-Amz-Checksum-Algorithm")
|
||||
|
||||
// Aborts the multipart upload in progress, if the
|
||||
// function returns any error, since we do not resume
|
||||
// we should purge the parts which have been uploaded
|
||||
// to relinquish storage space.
|
||||
defer func() {
|
||||
if err != nil {
|
||||
c.abortMultipartUpload(ctx, bucketName, objectName, uploadID)
|
||||
}
|
||||
}()
|
||||
|
||||
// Total data read and written to server. should be equal to 'size' at the end of the call.
|
||||
var totalUploadedSize int64
|
||||
|
||||
// Complete multipart upload.
|
||||
var complMultipartUpload completeMultipartUpload
|
||||
|
||||
// Declare a channel that sends the next part number to be uploaded.
|
||||
uploadPartsCh := make(chan uploadPartReq)
|
||||
|
||||
// Declare a channel that sends back the response of a part upload.
|
||||
uploadedPartsCh := make(chan uploadedPartRes)
|
||||
|
||||
// Used for readability, lastPartNumber is always totalPartsCount.
|
||||
lastPartNumber := totalPartsCount
|
||||
|
||||
partitionCtx, partitionCancel := context.WithCancel(ctx)
|
||||
defer partitionCancel()
|
||||
// Send each part number to the channel to be processed.
|
||||
go func() {
|
||||
defer close(uploadPartsCh)
|
||||
|
||||
for p := 1; p <= totalPartsCount; p++ {
|
||||
select {
|
||||
case <-partitionCtx.Done():
|
||||
return
|
||||
case uploadPartsCh <- uploadPartReq{PartNum: p}:
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
// Receive each part number from the channel allowing three parallel uploads.
|
||||
for w := 1; w <= opts.getNumThreads(); w++ {
|
||||
go func(partSize int64) {
|
||||
for {
|
||||
var uploadReq uploadPartReq
|
||||
var ok bool
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case uploadReq, ok = <-uploadPartsCh:
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
// Each worker will draw from the part channel and upload in parallel.
|
||||
}
|
||||
|
||||
// If partNumber was not uploaded we calculate the missing
|
||||
// part offset and size. For all other part numbers we
|
||||
// calculate offset based on multiples of partSize.
|
||||
readOffset := int64(uploadReq.PartNum-1) * partSize
|
||||
|
||||
// As a special case if partNumber is lastPartNumber, we
|
||||
// calculate the offset based on the last part size.
|
||||
if uploadReq.PartNum == lastPartNumber {
|
||||
readOffset = size - lastPartSize
|
||||
partSize = lastPartSize
|
||||
}
|
||||
|
||||
sectionReader := newHook(io.NewSectionReader(reader, readOffset, partSize), opts.Progress)
|
||||
trailer := make(http.Header, 1)
|
||||
if withChecksum {
|
||||
crc := opts.AutoChecksum.Hasher()
|
||||
trailer.Set(opts.AutoChecksum.Key(), base64.StdEncoding.EncodeToString(crc.Sum(nil)))
|
||||
sectionReader = newHashReaderWrapper(sectionReader, crc, func(hash []byte) {
|
||||
trailer.Set(opts.AutoChecksum.Key(), base64.StdEncoding.EncodeToString(hash))
|
||||
})
|
||||
}
|
||||
|
||||
// Proceed to upload the part.
|
||||
p := uploadPartParams{
|
||||
bucketName: bucketName,
|
||||
objectName: objectName,
|
||||
uploadID: uploadID,
|
||||
reader: sectionReader,
|
||||
partNumber: uploadReq.PartNum,
|
||||
size: partSize,
|
||||
sse: opts.ServerSideEncryption,
|
||||
streamSha256: !opts.DisableContentSha256,
|
||||
sha256Hex: "",
|
||||
trailer: trailer,
|
||||
}
|
||||
objPart, err := c.uploadPart(ctx, p)
|
||||
if err != nil {
|
||||
uploadedPartsCh <- uploadedPartRes{
|
||||
Error: err,
|
||||
}
|
||||
// Exit the goroutine.
|
||||
return
|
||||
}
|
||||
|
||||
// Save successfully uploaded part metadata.
|
||||
uploadReq.Part = objPart
|
||||
|
||||
// Send successful part info through the channel.
|
||||
uploadedPartsCh <- uploadedPartRes{
|
||||
Size: objPart.Size,
|
||||
PartNum: uploadReq.PartNum,
|
||||
Part: uploadReq.Part,
|
||||
}
|
||||
}
|
||||
}(partSize)
|
||||
}
|
||||
|
||||
// Gather the responses as they occur and update any
|
||||
// progress bar.
|
||||
allParts := make([]ObjectPart, 0, totalPartsCount)
|
||||
for u := 1; u <= totalPartsCount; u++ {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return UploadInfo{}, ctx.Err()
|
||||
case uploadRes := <-uploadedPartsCh:
|
||||
if uploadRes.Error != nil {
|
||||
return UploadInfo{}, uploadRes.Error
|
||||
}
|
||||
allParts = append(allParts, uploadRes.Part)
|
||||
// Update the totalUploadedSize.
|
||||
totalUploadedSize += uploadRes.Size
|
||||
complMultipartUpload.Parts = append(complMultipartUpload.Parts, CompletePart{
|
||||
ETag: uploadRes.Part.ETag,
|
||||
PartNumber: uploadRes.Part.PartNumber,
|
||||
ChecksumCRC32: uploadRes.Part.ChecksumCRC32,
|
||||
ChecksumCRC32C: uploadRes.Part.ChecksumCRC32C,
|
||||
ChecksumSHA1: uploadRes.Part.ChecksumSHA1,
|
||||
ChecksumSHA256: uploadRes.Part.ChecksumSHA256,
|
||||
ChecksumCRC64NVME: uploadRes.Part.ChecksumCRC64NVME,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Verify if we uploaded all the data.
|
||||
if totalUploadedSize != size {
|
||||
return UploadInfo{}, errUnexpectedEOF(totalUploadedSize, size, bucketName, objectName)
|
||||
}
|
||||
|
||||
// Sort all completed parts.
|
||||
sort.Sort(completedParts(complMultipartUpload.Parts))
|
||||
|
||||
opts = PutObjectOptions{
|
||||
ServerSideEncryption: opts.ServerSideEncryption,
|
||||
AutoChecksum: opts.AutoChecksum,
|
||||
}
|
||||
if withChecksum {
|
||||
applyAutoChecksum(&opts, allParts)
|
||||
}
|
||||
|
||||
uploadInfo, err := c.completeMultipartUpload(ctx, bucketName, objectName, uploadID, complMultipartUpload, opts)
|
||||
if err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
|
||||
uploadInfo.Size = totalUploadedSize
|
||||
return uploadInfo, nil
|
||||
}
|
||||
|
||||
func (c *Client) putObjectMultipartStreamOptionalChecksum(ctx context.Context, bucketName, objectName string,
|
||||
reader io.Reader, size int64, opts PutObjectOptions,
|
||||
) (info UploadInfo, err error) {
|
||||
// Input validation.
|
||||
if err = s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
if err = s3utils.CheckValidObjectName(objectName); err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
|
||||
if opts.Checksum.IsSet() {
|
||||
opts.AutoChecksum = opts.Checksum
|
||||
opts.SendContentMd5 = false
|
||||
}
|
||||
|
||||
if !opts.SendContentMd5 {
|
||||
addAutoChecksumHeaders(&opts)
|
||||
}
|
||||
|
||||
// Calculate the optimal parts info for a given size.
|
||||
totalPartsCount, partSize, lastPartSize, err := OptimalPartInfo(size, opts.PartSize)
|
||||
if err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
// Initiates a new multipart request
|
||||
uploadID, err := c.newUploadID(ctx, bucketName, objectName, opts)
|
||||
if err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
delete(opts.UserMetadata, "X-Amz-Checksum-Algorithm")
|
||||
|
||||
// Aborts the multipart upload if the function returns
|
||||
// any error, since we do not resume we should purge
|
||||
// the parts which have been uploaded to relinquish
|
||||
// storage space.
|
||||
defer func() {
|
||||
if err != nil {
|
||||
c.abortMultipartUpload(ctx, bucketName, objectName, uploadID)
|
||||
}
|
||||
}()
|
||||
|
||||
// Create checksums
|
||||
// CRC32C is ~50% faster on AMD64 @ 30GB/s
|
||||
customHeader := make(http.Header)
|
||||
crc := opts.AutoChecksum.Hasher()
|
||||
md5Hash := c.md5Hasher()
|
||||
defer md5Hash.Close()
|
||||
|
||||
// Total data read and written to server. should be equal to 'size' at the end of the call.
|
||||
var totalUploadedSize int64
|
||||
|
||||
// Initialize parts uploaded map.
|
||||
partsInfo := make(map[int]ObjectPart)
|
||||
|
||||
// Create a buffer.
|
||||
buf := make([]byte, partSize)
|
||||
|
||||
// Avoid declaring variables in the for loop
|
||||
var md5Base64 string
|
||||
|
||||
// Part number always starts with '1'.
|
||||
var partNumber int
|
||||
for partNumber = 1; partNumber <= totalPartsCount; partNumber++ {
|
||||
|
||||
// Proceed to upload the part.
|
||||
if partNumber == totalPartsCount {
|
||||
partSize = lastPartSize
|
||||
}
|
||||
|
||||
length, rerr := readFull(reader, buf)
|
||||
if rerr == io.EOF && partNumber > 1 {
|
||||
break
|
||||
}
|
||||
|
||||
if rerr != nil && rerr != io.ErrUnexpectedEOF && err != io.EOF {
|
||||
return UploadInfo{}, rerr
|
||||
}
|
||||
|
||||
// Calculate md5sum.
|
||||
if opts.SendContentMd5 {
|
||||
md5Hash.Reset()
|
||||
md5Hash.Write(buf[:length])
|
||||
md5Base64 = base64.StdEncoding.EncodeToString(md5Hash.Sum(nil))
|
||||
} else {
|
||||
// Add CRC32C instead.
|
||||
crc.Reset()
|
||||
crc.Write(buf[:length])
|
||||
cSum := crc.Sum(nil)
|
||||
customHeader.Set(opts.AutoChecksum.KeyCapitalized(), base64.StdEncoding.EncodeToString(cSum))
|
||||
}
|
||||
|
||||
// Update progress reader appropriately to the latest offset
|
||||
// as we read from the source.
|
||||
hooked := newHook(bytes.NewReader(buf[:length]), opts.Progress)
|
||||
p := uploadPartParams{bucketName: bucketName, objectName: objectName, uploadID: uploadID, reader: hooked, partNumber: partNumber, md5Base64: md5Base64, size: partSize, sse: opts.ServerSideEncryption, streamSha256: !opts.DisableContentSha256, customHeader: customHeader}
|
||||
objPart, uerr := c.uploadPart(ctx, p)
|
||||
if uerr != nil {
|
||||
return UploadInfo{}, uerr
|
||||
}
|
||||
|
||||
// Save successfully uploaded part metadata.
|
||||
partsInfo[partNumber] = objPart
|
||||
|
||||
// Save successfully uploaded size.
|
||||
totalUploadedSize += partSize
|
||||
}
|
||||
|
||||
// Verify if we uploaded all the data.
|
||||
if size > 0 {
|
||||
if totalUploadedSize != size {
|
||||
return UploadInfo{}, errUnexpectedEOF(totalUploadedSize, size, bucketName, objectName)
|
||||
}
|
||||
}
|
||||
|
||||
// Complete multipart upload.
|
||||
var complMultipartUpload completeMultipartUpload
|
||||
|
||||
// Loop over total uploaded parts to save them in
|
||||
// Parts array before completing the multipart request.
|
||||
allParts := make([]ObjectPart, 0, len(partsInfo))
|
||||
for i := 1; i < partNumber; i++ {
|
||||
part, ok := partsInfo[i]
|
||||
if !ok {
|
||||
return UploadInfo{}, errInvalidArgument(fmt.Sprintf("Missing part number %d", i))
|
||||
}
|
||||
allParts = append(allParts, part)
|
||||
complMultipartUpload.Parts = append(complMultipartUpload.Parts, CompletePart{
|
||||
ETag: part.ETag,
|
||||
PartNumber: part.PartNumber,
|
||||
ChecksumCRC32: part.ChecksumCRC32,
|
||||
ChecksumCRC32C: part.ChecksumCRC32C,
|
||||
ChecksumSHA1: part.ChecksumSHA1,
|
||||
ChecksumSHA256: part.ChecksumSHA256,
|
||||
ChecksumCRC64NVME: part.ChecksumCRC64NVME,
|
||||
})
|
||||
}
|
||||
|
||||
// Sort all completed parts.
|
||||
sort.Sort(completedParts(complMultipartUpload.Parts))
|
||||
|
||||
opts = PutObjectOptions{
|
||||
ServerSideEncryption: opts.ServerSideEncryption,
|
||||
AutoChecksum: opts.AutoChecksum,
|
||||
}
|
||||
applyAutoChecksum(&opts, allParts)
|
||||
uploadInfo, err := c.completeMultipartUpload(ctx, bucketName, objectName, uploadID, complMultipartUpload, opts)
|
||||
if err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
|
||||
uploadInfo.Size = totalUploadedSize
|
||||
return uploadInfo, nil
|
||||
}
|
||||
|
||||
// putObjectMultipartStreamParallel uploads opts.NumThreads parts in parallel.
|
||||
// This is expected to take opts.PartSize * opts.NumThreads * (GOGC / 100) bytes of buffer.
|
||||
func (c *Client) putObjectMultipartStreamParallel(ctx context.Context, bucketName, objectName string,
|
||||
reader io.Reader, opts PutObjectOptions,
|
||||
) (info UploadInfo, err error) {
|
||||
// Input validation.
|
||||
if err = s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
|
||||
if err = s3utils.CheckValidObjectName(objectName); err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
if opts.Checksum.IsSet() {
|
||||
opts.SendContentMd5 = false
|
||||
opts.AutoChecksum = opts.Checksum
|
||||
}
|
||||
if !opts.SendContentMd5 {
|
||||
addAutoChecksumHeaders(&opts)
|
||||
}
|
||||
|
||||
// Cancel all when an error occurs.
|
||||
ctx, cancel := context.WithCancel(ctx)
|
||||
defer cancel()
|
||||
|
||||
// Calculate the optimal parts info for a given size.
|
||||
totalPartsCount, partSize, _, err := OptimalPartInfo(-1, opts.PartSize)
|
||||
if err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
|
||||
// Initiates a new multipart request
|
||||
uploadID, err := c.newUploadID(ctx, bucketName, objectName, opts)
|
||||
if err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
delete(opts.UserMetadata, "X-Amz-Checksum-Algorithm")
|
||||
|
||||
// Aborts the multipart upload if the function returns
|
||||
// any error, since we do not resume we should purge
|
||||
// the parts which have been uploaded to relinquish
|
||||
// storage space.
|
||||
defer func() {
|
||||
if err != nil {
|
||||
c.abortMultipartUpload(ctx, bucketName, objectName, uploadID)
|
||||
}
|
||||
}()
|
||||
|
||||
// Create checksums
|
||||
// CRC32C is ~50% faster on AMD64 @ 30GB/s
|
||||
crc := opts.AutoChecksum.Hasher()
|
||||
|
||||
// Total data read and written to server. should be equal to 'size' at the end of the call.
|
||||
var totalUploadedSize int64
|
||||
|
||||
// Initialize parts uploaded map.
|
||||
partsInfo := make(map[int]ObjectPart)
|
||||
|
||||
// Create a buffer.
|
||||
nBuffers := int64(opts.NumThreads)
|
||||
bufs := make(chan []byte, nBuffers)
|
||||
all := make([]byte, nBuffers*partSize)
|
||||
for i := int64(0); i < nBuffers; i++ {
|
||||
bufs <- all[i*partSize : i*partSize+partSize]
|
||||
}
|
||||
|
||||
var wg sync.WaitGroup
|
||||
var mu sync.Mutex
|
||||
errCh := make(chan error, opts.NumThreads)
|
||||
|
||||
reader = newHook(reader, opts.Progress)
|
||||
|
||||
// Part number always starts with '1'.
|
||||
var partNumber int
|
||||
for partNumber = 1; partNumber <= totalPartsCount; partNumber++ {
|
||||
// Proceed to upload the part.
|
||||
var buf []byte
|
||||
select {
|
||||
case buf = <-bufs:
|
||||
case err = <-errCh:
|
||||
cancel()
|
||||
wg.Wait()
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
|
||||
if int64(len(buf)) != partSize {
|
||||
return UploadInfo{}, fmt.Errorf("read buffer < %d than expected partSize: %d", len(buf), partSize)
|
||||
}
|
||||
|
||||
length, rerr := readFull(reader, buf)
|
||||
if rerr == io.EOF && partNumber > 1 {
|
||||
// Done
|
||||
break
|
||||
}
|
||||
|
||||
if rerr != nil && rerr != io.ErrUnexpectedEOF && err != io.EOF {
|
||||
cancel()
|
||||
wg.Wait()
|
||||
return UploadInfo{}, rerr
|
||||
}
|
||||
|
||||
// Calculate md5sum.
|
||||
customHeader := make(http.Header)
|
||||
if !opts.SendContentMd5 {
|
||||
// Add Checksum instead.
|
||||
crc.Reset()
|
||||
crc.Write(buf[:length])
|
||||
cSum := crc.Sum(nil)
|
||||
customHeader.Set(opts.AutoChecksum.Key(), base64.StdEncoding.EncodeToString(cSum))
|
||||
}
|
||||
|
||||
wg.Add(1)
|
||||
go func(partNumber int) {
|
||||
// Avoid declaring variables in the for loop
|
||||
var md5Base64 string
|
||||
|
||||
if opts.SendContentMd5 {
|
||||
md5Hash := c.md5Hasher()
|
||||
md5Hash.Write(buf[:length])
|
||||
md5Base64 = base64.StdEncoding.EncodeToString(md5Hash.Sum(nil))
|
||||
md5Hash.Close()
|
||||
}
|
||||
|
||||
defer wg.Done()
|
||||
p := uploadPartParams{
|
||||
bucketName: bucketName,
|
||||
objectName: objectName,
|
||||
uploadID: uploadID,
|
||||
reader: bytes.NewReader(buf[:length]),
|
||||
partNumber: partNumber,
|
||||
md5Base64: md5Base64,
|
||||
size: int64(length),
|
||||
sse: opts.ServerSideEncryption,
|
||||
streamSha256: !opts.DisableContentSha256,
|
||||
customHeader: customHeader,
|
||||
}
|
||||
objPart, uerr := c.uploadPart(ctx, p)
|
||||
if uerr != nil {
|
||||
errCh <- uerr
|
||||
return
|
||||
}
|
||||
|
||||
// Save successfully uploaded part metadata.
|
||||
mu.Lock()
|
||||
partsInfo[partNumber] = objPart
|
||||
mu.Unlock()
|
||||
|
||||
// Send buffer back so it can be reused.
|
||||
bufs <- buf
|
||||
}(partNumber)
|
||||
|
||||
// Save successfully uploaded size.
|
||||
totalUploadedSize += int64(length)
|
||||
}
|
||||
wg.Wait()
|
||||
|
||||
// Collect any error
|
||||
select {
|
||||
case err = <-errCh:
|
||||
return UploadInfo{}, err
|
||||
default:
|
||||
}
|
||||
|
||||
// Complete multipart upload.
|
||||
var complMultipartUpload completeMultipartUpload
|
||||
|
||||
// Loop over total uploaded parts to save them in
|
||||
// Parts array before completing the multipart request.
|
||||
allParts := make([]ObjectPart, 0, len(partsInfo))
|
||||
for i := 1; i < partNumber; i++ {
|
||||
part, ok := partsInfo[i]
|
||||
if !ok {
|
||||
return UploadInfo{}, errInvalidArgument(fmt.Sprintf("Missing part number %d", i))
|
||||
}
|
||||
allParts = append(allParts, part)
|
||||
complMultipartUpload.Parts = append(complMultipartUpload.Parts, CompletePart{
|
||||
ETag: part.ETag,
|
||||
PartNumber: part.PartNumber,
|
||||
ChecksumCRC32: part.ChecksumCRC32,
|
||||
ChecksumCRC32C: part.ChecksumCRC32C,
|
||||
ChecksumSHA1: part.ChecksumSHA1,
|
||||
ChecksumSHA256: part.ChecksumSHA256,
|
||||
ChecksumCRC64NVME: part.ChecksumCRC64NVME,
|
||||
})
|
||||
}
|
||||
|
||||
// Sort all completed parts.
|
||||
sort.Sort(completedParts(complMultipartUpload.Parts))
|
||||
|
||||
opts = PutObjectOptions{
|
||||
ServerSideEncryption: opts.ServerSideEncryption,
|
||||
AutoChecksum: opts.AutoChecksum,
|
||||
}
|
||||
applyAutoChecksum(&opts, allParts)
|
||||
|
||||
uploadInfo, err := c.completeMultipartUpload(ctx, bucketName, objectName, uploadID, complMultipartUpload, opts)
|
||||
if err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
|
||||
uploadInfo.Size = totalUploadedSize
|
||||
return uploadInfo, nil
|
||||
}
|
||||
|
||||
// putObject special function used Google Cloud Storage. This special function
|
||||
// is used for Google Cloud Storage since Google's multipart API is not S3 compatible.
|
||||
func (c *Client) putObject(ctx context.Context, bucketName, objectName string, reader io.Reader, size int64, opts PutObjectOptions) (info UploadInfo, err error) {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
if err := s3utils.CheckValidObjectName(objectName); err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
|
||||
// Size -1 is only supported on Google Cloud Storage, we error
|
||||
// out in all other situations.
|
||||
if size < 0 && !s3utils.IsGoogleEndpoint(*c.endpointURL) {
|
||||
return UploadInfo{}, errEntityTooSmall(size, bucketName, objectName)
|
||||
}
|
||||
|
||||
if opts.SendContentMd5 && s3utils.IsGoogleEndpoint(*c.endpointURL) && size < 0 {
|
||||
return UploadInfo{}, errInvalidArgument("MD5Sum cannot be calculated with size '-1'")
|
||||
}
|
||||
if opts.Checksum.IsSet() {
|
||||
opts.SendContentMd5 = false
|
||||
}
|
||||
|
||||
var readSeeker io.Seeker
|
||||
if size > 0 {
|
||||
if isReadAt(reader) && !isObject(reader) {
|
||||
seeker, ok := reader.(io.Seeker)
|
||||
if ok {
|
||||
offset, err := seeker.Seek(0, io.SeekCurrent)
|
||||
if err != nil {
|
||||
return UploadInfo{}, errInvalidArgument(err.Error())
|
||||
}
|
||||
reader = io.NewSectionReader(reader.(io.ReaderAt), offset, size)
|
||||
readSeeker = reader.(io.Seeker)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
var md5Base64 string
|
||||
if opts.SendContentMd5 {
|
||||
// Calculate md5sum.
|
||||
hash := c.md5Hasher()
|
||||
|
||||
if readSeeker != nil {
|
||||
if _, err := io.Copy(hash, reader); err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
// Seek back to beginning of io.NewSectionReader's offset.
|
||||
_, err = readSeeker.Seek(0, io.SeekStart)
|
||||
if err != nil {
|
||||
return UploadInfo{}, errInvalidArgument(err.Error())
|
||||
}
|
||||
} else {
|
||||
// Create a buffer.
|
||||
buf := make([]byte, size)
|
||||
|
||||
length, err := readFull(reader, buf)
|
||||
if err != nil && err != io.ErrUnexpectedEOF && err != io.EOF {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
|
||||
hash.Write(buf[:length])
|
||||
reader = bytes.NewReader(buf[:length])
|
||||
}
|
||||
|
||||
md5Base64 = base64.StdEncoding.EncodeToString(hash.Sum(nil))
|
||||
hash.Close()
|
||||
}
|
||||
|
||||
// Update progress reader appropriately to the latest offset as we
|
||||
// read from the source.
|
||||
progressReader := newHook(reader, opts.Progress)
|
||||
|
||||
// This function does not calculate sha256 and md5sum for payload.
|
||||
// Execute put object.
|
||||
return c.putObjectDo(ctx, bucketName, objectName, progressReader, md5Base64, "", size, opts)
|
||||
}
|
||||
|
||||
// putObjectDo - executes the put object http operation.
|
||||
// NOTE: You must have WRITE permissions on a bucket to add an object to it.
|
||||
func (c *Client) putObjectDo(ctx context.Context, bucketName, objectName string, reader io.Reader, md5Base64, sha256Hex string, size int64, opts PutObjectOptions) (UploadInfo, error) {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
if err := s3utils.CheckValidObjectName(objectName); err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
// Set headers.
|
||||
customHeader := opts.Header()
|
||||
|
||||
// Populate request metadata.
|
||||
reqMetadata := requestMetadata{
|
||||
bucketName: bucketName,
|
||||
objectName: objectName,
|
||||
customHeader: customHeader,
|
||||
contentBody: reader,
|
||||
contentLength: size,
|
||||
contentMD5Base64: md5Base64,
|
||||
contentSHA256Hex: sha256Hex,
|
||||
streamSha256: !opts.DisableContentSha256,
|
||||
}
|
||||
// Add CRC when client supports it, MD5 is not set, not Google and we don't add SHA256 to chunks.
|
||||
addCrc := c.trailingHeaderSupport && md5Base64 == "" && !s3utils.IsGoogleEndpoint(*c.endpointURL) && (opts.DisableContentSha256 || c.secure)
|
||||
if opts.Checksum.IsSet() {
|
||||
reqMetadata.addCrc = &opts.Checksum
|
||||
} else if addCrc {
|
||||
// If user has added checksums, don't add them ourselves.
|
||||
for k := range opts.UserMetadata {
|
||||
if strings.HasPrefix(strings.ToLower(k), "x-amz-checksum-") {
|
||||
addCrc = false
|
||||
}
|
||||
}
|
||||
if addCrc {
|
||||
opts.AutoChecksum.SetDefault(ChecksumCRC32C)
|
||||
reqMetadata.addCrc = &opts.AutoChecksum
|
||||
}
|
||||
}
|
||||
|
||||
if opts.Internal.SourceVersionID != "" {
|
||||
if opts.Internal.SourceVersionID != nullVersionID {
|
||||
if _, err := uuid.Parse(opts.Internal.SourceVersionID); err != nil {
|
||||
return UploadInfo{}, errInvalidArgument(err.Error())
|
||||
}
|
||||
}
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("versionId", opts.Internal.SourceVersionID)
|
||||
reqMetadata.queryValues = urlValues
|
||||
}
|
||||
|
||||
// Execute PUT an objectName.
|
||||
resp, err := c.executeMethod(ctx, http.MethodPut, reqMetadata)
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
if resp != nil {
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return UploadInfo{}, httpRespToErrorResponse(resp, bucketName, objectName)
|
||||
}
|
||||
}
|
||||
|
||||
// extract lifecycle expiry date and rule ID
|
||||
expTime, ruleID := amzExpirationToExpiryDateRuleID(resp.Header.Get(amzExpiration))
|
||||
h := resp.Header
|
||||
return UploadInfo{
|
||||
Bucket: bucketName,
|
||||
Key: objectName,
|
||||
ETag: trimEtag(h.Get("ETag")),
|
||||
VersionID: h.Get(amzVersionID),
|
||||
Size: size,
|
||||
Expiration: expTime,
|
||||
ExpirationRuleID: ruleID,
|
||||
|
||||
// Checksum values
|
||||
ChecksumCRC32: h.Get(ChecksumCRC32.Key()),
|
||||
ChecksumCRC32C: h.Get(ChecksumCRC32C.Key()),
|
||||
ChecksumSHA1: h.Get(ChecksumSHA1.Key()),
|
||||
ChecksumSHA256: h.Get(ChecksumSHA256.Key()),
|
||||
ChecksumCRC64NVME: h.Get(ChecksumCRC64NVME.Key()),
|
||||
}, nil
|
||||
}
|
||||
+507
@@ -0,0 +1,507 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2015-2017 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"github.com/minio/minio-go/v7/pkg/encrypt"
|
||||
"github.com/minio/minio-go/v7/pkg/s3utils"
|
||||
"golang.org/x/net/http/httpguts"
|
||||
)
|
||||
|
||||
// ReplicationStatus represents replication status of object
|
||||
type ReplicationStatus string
|
||||
|
||||
const (
|
||||
// ReplicationStatusPending indicates replication is pending
|
||||
ReplicationStatusPending ReplicationStatus = "PENDING"
|
||||
// ReplicationStatusComplete indicates replication completed ok
|
||||
ReplicationStatusComplete ReplicationStatus = "COMPLETED"
|
||||
// ReplicationStatusFailed indicates replication failed
|
||||
ReplicationStatusFailed ReplicationStatus = "FAILED"
|
||||
// ReplicationStatusReplica indicates object is a replica of a source
|
||||
ReplicationStatusReplica ReplicationStatus = "REPLICA"
|
||||
// ReplicationStatusReplicaEdge indicates object is a replica of a edge source
|
||||
ReplicationStatusReplicaEdge ReplicationStatus = "REPLICA-EDGE"
|
||||
)
|
||||
|
||||
// Empty returns true if no replication status set.
|
||||
func (r ReplicationStatus) Empty() bool {
|
||||
return r == ""
|
||||
}
|
||||
|
||||
// AdvancedPutOptions for internal use - to be utilized by replication, ILM transition
|
||||
// implementation on MinIO server
|
||||
type AdvancedPutOptions struct {
|
||||
SourceVersionID string
|
||||
SourceETag string
|
||||
ReplicationStatus ReplicationStatus
|
||||
SourceMTime time.Time
|
||||
ReplicationRequest bool
|
||||
RetentionTimestamp time.Time
|
||||
TaggingTimestamp time.Time
|
||||
LegalholdTimestamp time.Time
|
||||
ReplicationValidityCheck bool
|
||||
}
|
||||
|
||||
// PutObjectOptions represents options specified by user for PutObject call
|
||||
type PutObjectOptions struct {
|
||||
UserMetadata map[string]string
|
||||
UserTags map[string]string
|
||||
Progress io.Reader
|
||||
ContentType string
|
||||
ContentEncoding string
|
||||
ContentDisposition string
|
||||
ContentLanguage string
|
||||
CacheControl string
|
||||
Expires time.Time
|
||||
Mode RetentionMode
|
||||
RetainUntilDate time.Time
|
||||
ServerSideEncryption encrypt.ServerSide
|
||||
NumThreads uint
|
||||
StorageClass string
|
||||
WebsiteRedirectLocation string
|
||||
PartSize uint64
|
||||
LegalHold LegalHoldStatus
|
||||
SendContentMd5 bool
|
||||
DisableContentSha256 bool
|
||||
DisableMultipart bool
|
||||
|
||||
// AutoChecksum is the type of checksum that will be added if no other checksum is added,
|
||||
// like MD5 or SHA256 streaming checksum, and it is feasible for the upload type.
|
||||
// If none is specified CRC32C is used, since it is generally the fastest.
|
||||
AutoChecksum ChecksumType
|
||||
|
||||
// Checksum will force a checksum of the specific type.
|
||||
// This requires that the client was created with "TrailingHeaders:true" option,
|
||||
// and that the destination server supports it.
|
||||
// Unavailable with V2 signatures & Google endpoints.
|
||||
// This will disable content MD5 checksums if set.
|
||||
Checksum ChecksumType
|
||||
|
||||
// ConcurrentStreamParts will create NumThreads buffers of PartSize bytes,
|
||||
// fill them serially and upload them in parallel.
|
||||
// This can be used for faster uploads on non-seekable or slow-to-seek input.
|
||||
ConcurrentStreamParts bool
|
||||
Internal AdvancedPutOptions
|
||||
|
||||
customHeaders http.Header
|
||||
}
|
||||
|
||||
// SetMatchETag if etag matches while PUT MinIO returns an error
|
||||
// this is a MinIO specific extension to support optimistic locking
|
||||
// semantics.
|
||||
func (opts *PutObjectOptions) SetMatchETag(etag string) {
|
||||
if opts.customHeaders == nil {
|
||||
opts.customHeaders = http.Header{}
|
||||
}
|
||||
if etag == "*" {
|
||||
opts.customHeaders.Set("If-Match", "*")
|
||||
} else {
|
||||
opts.customHeaders.Set("If-Match", "\""+etag+"\"")
|
||||
}
|
||||
}
|
||||
|
||||
// SetMatchETagExcept if etag does not match while PUT MinIO returns an
|
||||
// error this is a MinIO specific extension to support optimistic locking
|
||||
// semantics.
|
||||
func (opts *PutObjectOptions) SetMatchETagExcept(etag string) {
|
||||
if opts.customHeaders == nil {
|
||||
opts.customHeaders = http.Header{}
|
||||
}
|
||||
if etag == "*" {
|
||||
opts.customHeaders.Set("If-None-Match", "*")
|
||||
} else {
|
||||
opts.customHeaders.Set("If-None-Match", "\""+etag+"\"")
|
||||
}
|
||||
}
|
||||
|
||||
// getNumThreads - gets the number of threads to be used in the multipart
|
||||
// put object operation
|
||||
func (opts PutObjectOptions) getNumThreads() (numThreads int) {
|
||||
if opts.NumThreads > 0 {
|
||||
numThreads = int(opts.NumThreads)
|
||||
} else {
|
||||
numThreads = totalWorkers
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// Header - constructs the headers from metadata entered by user in
|
||||
// PutObjectOptions struct
|
||||
func (opts PutObjectOptions) Header() (header http.Header) {
|
||||
header = make(http.Header)
|
||||
|
||||
contentType := opts.ContentType
|
||||
if contentType == "" {
|
||||
contentType = "application/octet-stream"
|
||||
}
|
||||
header.Set("Content-Type", contentType)
|
||||
|
||||
if opts.ContentEncoding != "" {
|
||||
header.Set("Content-Encoding", opts.ContentEncoding)
|
||||
}
|
||||
if opts.ContentDisposition != "" {
|
||||
header.Set("Content-Disposition", opts.ContentDisposition)
|
||||
}
|
||||
if opts.ContentLanguage != "" {
|
||||
header.Set("Content-Language", opts.ContentLanguage)
|
||||
}
|
||||
if opts.CacheControl != "" {
|
||||
header.Set("Cache-Control", opts.CacheControl)
|
||||
}
|
||||
|
||||
if !opts.Expires.IsZero() {
|
||||
header.Set("Expires", opts.Expires.UTC().Format(http.TimeFormat))
|
||||
}
|
||||
|
||||
if opts.Mode != "" {
|
||||
header.Set(amzLockMode, opts.Mode.String())
|
||||
}
|
||||
|
||||
if !opts.RetainUntilDate.IsZero() {
|
||||
header.Set("X-Amz-Object-Lock-Retain-Until-Date", opts.RetainUntilDate.Format(time.RFC3339))
|
||||
}
|
||||
|
||||
if opts.LegalHold != "" {
|
||||
header.Set(amzLegalHoldHeader, opts.LegalHold.String())
|
||||
}
|
||||
|
||||
if opts.ServerSideEncryption != nil {
|
||||
opts.ServerSideEncryption.Marshal(header)
|
||||
}
|
||||
|
||||
if opts.StorageClass != "" {
|
||||
header.Set(amzStorageClass, opts.StorageClass)
|
||||
}
|
||||
|
||||
if opts.WebsiteRedirectLocation != "" {
|
||||
header.Set(amzWebsiteRedirectLocation, opts.WebsiteRedirectLocation)
|
||||
}
|
||||
|
||||
if !opts.Internal.ReplicationStatus.Empty() {
|
||||
header.Set(amzBucketReplicationStatus, string(opts.Internal.ReplicationStatus))
|
||||
}
|
||||
if !opts.Internal.SourceMTime.IsZero() {
|
||||
header.Set(minIOBucketSourceMTime, opts.Internal.SourceMTime.Format(time.RFC3339Nano))
|
||||
}
|
||||
if opts.Internal.SourceETag != "" {
|
||||
header.Set(minIOBucketSourceETag, opts.Internal.SourceETag)
|
||||
}
|
||||
if opts.Internal.ReplicationRequest {
|
||||
header.Set(minIOBucketReplicationRequest, "true")
|
||||
}
|
||||
if opts.Internal.ReplicationValidityCheck {
|
||||
header.Set(minIOBucketReplicationCheck, "true")
|
||||
}
|
||||
if !opts.Internal.LegalholdTimestamp.IsZero() {
|
||||
header.Set(minIOBucketReplicationObjectLegalHoldTimestamp, opts.Internal.LegalholdTimestamp.Format(time.RFC3339Nano))
|
||||
}
|
||||
if !opts.Internal.RetentionTimestamp.IsZero() {
|
||||
header.Set(minIOBucketReplicationObjectRetentionTimestamp, opts.Internal.RetentionTimestamp.Format(time.RFC3339Nano))
|
||||
}
|
||||
if !opts.Internal.TaggingTimestamp.IsZero() {
|
||||
header.Set(minIOBucketReplicationTaggingTimestamp, opts.Internal.TaggingTimestamp.Format(time.RFC3339Nano))
|
||||
}
|
||||
|
||||
if len(opts.UserTags) != 0 {
|
||||
header.Set(amzTaggingHeader, s3utils.TagEncode(opts.UserTags))
|
||||
}
|
||||
|
||||
for k, v := range opts.UserMetadata {
|
||||
if isAmzHeader(k) || isStandardHeader(k) || isStorageClassHeader(k) || isMinioHeader(k) {
|
||||
header.Set(k, v)
|
||||
} else {
|
||||
header.Set("x-amz-meta-"+k, v)
|
||||
}
|
||||
}
|
||||
|
||||
// set any other additional custom headers.
|
||||
for k, v := range opts.customHeaders {
|
||||
header[k] = v
|
||||
}
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
// validate() checks if the UserMetadata map has standard headers or and raises an error if so.
|
||||
func (opts PutObjectOptions) validate(c *Client) (err error) {
|
||||
for k, v := range opts.UserMetadata {
|
||||
if !httpguts.ValidHeaderFieldName(k) || isStandardHeader(k) || isSSEHeader(k) || isStorageClassHeader(k) || isMinioHeader(k) {
|
||||
return errInvalidArgument(k + " unsupported user defined metadata name")
|
||||
}
|
||||
if !httpguts.ValidHeaderFieldValue(v) {
|
||||
return errInvalidArgument(v + " unsupported user defined metadata value")
|
||||
}
|
||||
}
|
||||
if opts.Mode != "" && !opts.Mode.IsValid() {
|
||||
return errInvalidArgument(opts.Mode.String() + " unsupported retention mode")
|
||||
}
|
||||
if opts.LegalHold != "" && !opts.LegalHold.IsValid() {
|
||||
return errInvalidArgument(opts.LegalHold.String() + " unsupported legal-hold status")
|
||||
}
|
||||
if opts.Checksum.IsSet() {
|
||||
switch {
|
||||
case !c.trailingHeaderSupport:
|
||||
return errInvalidArgument("Checksum requires Client with TrailingHeaders enabled")
|
||||
case c.overrideSignerType.IsV2():
|
||||
return errInvalidArgument("Checksum cannot be used with v2 signatures")
|
||||
case s3utils.IsGoogleEndpoint(*c.endpointURL):
|
||||
return errInvalidArgument("Checksum cannot be used with GCS endpoints")
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// completedParts is a collection of parts sortable by their part numbers.
|
||||
// used for sorting the uploaded parts before completing the multipart request.
|
||||
type completedParts []CompletePart
|
||||
|
||||
func (a completedParts) Len() int { return len(a) }
|
||||
func (a completedParts) Swap(i, j int) { a[i], a[j] = a[j], a[i] }
|
||||
func (a completedParts) Less(i, j int) bool { return a[i].PartNumber < a[j].PartNumber }
|
||||
|
||||
// PutObject creates an object in a bucket.
|
||||
//
|
||||
// You must have WRITE permissions on a bucket to create an object.
|
||||
//
|
||||
// - For size smaller than 16MiB PutObject automatically does a
|
||||
// single atomic PUT operation.
|
||||
//
|
||||
// - For size larger than 16MiB PutObject automatically does a
|
||||
// multipart upload operation.
|
||||
//
|
||||
// - For size input as -1 PutObject does a multipart Put operation
|
||||
// until input stream reaches EOF. Maximum object size that can
|
||||
// be uploaded through this operation will be 5TiB.
|
||||
//
|
||||
// WARNING: Passing down '-1' will use memory and these cannot
|
||||
// be reused for best outcomes for PutObject(), pass the size always.
|
||||
//
|
||||
// NOTE: Upon errors during upload multipart operation is entirely aborted.
|
||||
func (c *Client) PutObject(ctx context.Context, bucketName, objectName string, reader io.Reader, objectSize int64,
|
||||
opts PutObjectOptions,
|
||||
) (info UploadInfo, err error) {
|
||||
if objectSize < 0 && opts.DisableMultipart {
|
||||
return UploadInfo{}, errors.New("object size must be provided with disable multipart upload")
|
||||
}
|
||||
|
||||
err = opts.validate(c)
|
||||
if err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
|
||||
return c.putObjectCommon(ctx, bucketName, objectName, reader, objectSize, opts)
|
||||
}
|
||||
|
||||
func (c *Client) putObjectCommon(ctx context.Context, bucketName, objectName string, reader io.Reader, size int64, opts PutObjectOptions) (info UploadInfo, err error) {
|
||||
// Check for largest object size allowed.
|
||||
if size > int64(maxMultipartPutObjectSize) {
|
||||
return UploadInfo{}, errEntityTooLarge(size, maxMultipartPutObjectSize, bucketName, objectName)
|
||||
}
|
||||
opts.AutoChecksum.SetDefault(ChecksumCRC32C)
|
||||
|
||||
// NOTE: Streaming signature is not supported by GCS.
|
||||
if s3utils.IsGoogleEndpoint(*c.endpointURL) {
|
||||
return c.putObject(ctx, bucketName, objectName, reader, size, opts)
|
||||
}
|
||||
|
||||
partSize := opts.PartSize
|
||||
if opts.PartSize == 0 {
|
||||
partSize = minPartSize
|
||||
}
|
||||
|
||||
if c.overrideSignerType.IsV2() {
|
||||
if size >= 0 && size < int64(partSize) || opts.DisableMultipart {
|
||||
return c.putObject(ctx, bucketName, objectName, reader, size, opts)
|
||||
}
|
||||
return c.putObjectMultipart(ctx, bucketName, objectName, reader, size, opts)
|
||||
}
|
||||
|
||||
if size < 0 {
|
||||
if opts.DisableMultipart {
|
||||
return UploadInfo{}, errors.New("no length provided and multipart disabled")
|
||||
}
|
||||
if opts.ConcurrentStreamParts && opts.NumThreads > 1 {
|
||||
return c.putObjectMultipartStreamParallel(ctx, bucketName, objectName, reader, opts)
|
||||
}
|
||||
return c.putObjectMultipartStreamNoLength(ctx, bucketName, objectName, reader, opts)
|
||||
}
|
||||
|
||||
if size <= int64(partSize) || opts.DisableMultipart {
|
||||
return c.putObject(ctx, bucketName, objectName, reader, size, opts)
|
||||
}
|
||||
|
||||
return c.putObjectMultipartStream(ctx, bucketName, objectName, reader, size, opts)
|
||||
}
|
||||
|
||||
func (c *Client) putObjectMultipartStreamNoLength(ctx context.Context, bucketName, objectName string, reader io.Reader, opts PutObjectOptions) (info UploadInfo, err error) {
|
||||
// Input validation.
|
||||
if err = s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
if err = s3utils.CheckValidObjectName(objectName); err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
|
||||
// Total data read and written to server. should be equal to
|
||||
// 'size' at the end of the call.
|
||||
var totalUploadedSize int64
|
||||
|
||||
// Complete multipart upload.
|
||||
var complMultipartUpload completeMultipartUpload
|
||||
|
||||
// Calculate the optimal parts info for a given size.
|
||||
totalPartsCount, partSize, _, err := OptimalPartInfo(-1, opts.PartSize)
|
||||
if err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
|
||||
if opts.Checksum.IsSet() {
|
||||
opts.SendContentMd5 = false
|
||||
opts.AutoChecksum = opts.Checksum
|
||||
}
|
||||
if !opts.SendContentMd5 {
|
||||
addAutoChecksumHeaders(&opts)
|
||||
}
|
||||
|
||||
// Initiate a new multipart upload.
|
||||
uploadID, err := c.newUploadID(ctx, bucketName, objectName, opts)
|
||||
if err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
delete(opts.UserMetadata, "X-Amz-Checksum-Algorithm")
|
||||
|
||||
defer func() {
|
||||
if err != nil {
|
||||
c.abortMultipartUpload(ctx, bucketName, objectName, uploadID)
|
||||
}
|
||||
}()
|
||||
|
||||
// Part number always starts with '1'.
|
||||
partNumber := 1
|
||||
|
||||
// Initialize parts uploaded map.
|
||||
partsInfo := make(map[int]ObjectPart)
|
||||
|
||||
// Create a buffer.
|
||||
buf := make([]byte, partSize)
|
||||
|
||||
// Create checksums
|
||||
// CRC32C is ~50% faster on AMD64 @ 30GB/s
|
||||
customHeader := make(http.Header)
|
||||
crc := opts.AutoChecksum.Hasher()
|
||||
|
||||
for partNumber <= totalPartsCount {
|
||||
length, rerr := readFull(reader, buf)
|
||||
if rerr == io.EOF && partNumber > 1 {
|
||||
break
|
||||
}
|
||||
|
||||
if rerr != nil && rerr != io.ErrUnexpectedEOF && rerr != io.EOF {
|
||||
return UploadInfo{}, rerr
|
||||
}
|
||||
|
||||
var md5Base64 string
|
||||
if opts.SendContentMd5 {
|
||||
// Calculate md5sum.
|
||||
hash := c.md5Hasher()
|
||||
hash.Write(buf[:length])
|
||||
md5Base64 = base64.StdEncoding.EncodeToString(hash.Sum(nil))
|
||||
hash.Close()
|
||||
} else {
|
||||
crc.Reset()
|
||||
crc.Write(buf[:length])
|
||||
cSum := crc.Sum(nil)
|
||||
customHeader.Set(opts.AutoChecksum.Key(), base64.StdEncoding.EncodeToString(cSum))
|
||||
}
|
||||
|
||||
// Update progress reader appropriately to the latest offset
|
||||
// as we read from the source.
|
||||
rd := newHook(bytes.NewReader(buf[:length]), opts.Progress)
|
||||
|
||||
// Proceed to upload the part.
|
||||
p := uploadPartParams{bucketName: bucketName, objectName: objectName, uploadID: uploadID, reader: rd, partNumber: partNumber, md5Base64: md5Base64, size: int64(length), sse: opts.ServerSideEncryption, streamSha256: !opts.DisableContentSha256, customHeader: customHeader}
|
||||
objPart, uerr := c.uploadPart(ctx, p)
|
||||
if uerr != nil {
|
||||
return UploadInfo{}, uerr
|
||||
}
|
||||
|
||||
// Save successfully uploaded part metadata.
|
||||
partsInfo[partNumber] = objPart
|
||||
|
||||
// Save successfully uploaded size.
|
||||
totalUploadedSize += int64(length)
|
||||
|
||||
// Increment part number.
|
||||
partNumber++
|
||||
|
||||
// For unknown size, Read EOF we break away.
|
||||
// We do not have to upload till totalPartsCount.
|
||||
if rerr == io.EOF {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
// Loop over total uploaded parts to save them in
|
||||
// Parts array before completing the multipart request.
|
||||
allParts := make([]ObjectPart, 0, len(partsInfo))
|
||||
for i := 1; i < partNumber; i++ {
|
||||
part, ok := partsInfo[i]
|
||||
if !ok {
|
||||
return UploadInfo{}, errInvalidArgument(fmt.Sprintf("Missing part number %d", i))
|
||||
}
|
||||
allParts = append(allParts, part)
|
||||
complMultipartUpload.Parts = append(complMultipartUpload.Parts, CompletePart{
|
||||
ETag: part.ETag,
|
||||
PartNumber: part.PartNumber,
|
||||
ChecksumCRC32: part.ChecksumCRC32,
|
||||
ChecksumCRC32C: part.ChecksumCRC32C,
|
||||
ChecksumSHA1: part.ChecksumSHA1,
|
||||
ChecksumSHA256: part.ChecksumSHA256,
|
||||
ChecksumCRC64NVME: part.ChecksumCRC64NVME,
|
||||
})
|
||||
}
|
||||
|
||||
// Sort all completed parts.
|
||||
sort.Sort(completedParts(complMultipartUpload.Parts))
|
||||
|
||||
opts = PutObjectOptions{
|
||||
ServerSideEncryption: opts.ServerSideEncryption,
|
||||
AutoChecksum: opts.AutoChecksum,
|
||||
}
|
||||
applyAutoChecksum(&opts, allParts)
|
||||
|
||||
uploadInfo, err := c.completeMultipartUpload(ctx, bucketName, objectName, uploadID, complMultipartUpload, opts)
|
||||
if err != nil {
|
||||
return UploadInfo{}, err
|
||||
}
|
||||
|
||||
uploadInfo.Size = totalUploadedSize
|
||||
return uploadInfo, nil
|
||||
}
|
||||
+246
@@ -0,0 +1,246 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2021 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"bufio"
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/klauspost/compress/s2"
|
||||
)
|
||||
|
||||
// SnowballOptions contains options for PutObjectsSnowball calls.
|
||||
type SnowballOptions struct {
|
||||
// Opts is options applied to all objects.
|
||||
Opts PutObjectOptions
|
||||
|
||||
// Processing options:
|
||||
|
||||
// InMemory specifies that all objects should be collected in memory
|
||||
// before they are uploaded.
|
||||
// If false a temporary file will be created.
|
||||
InMemory bool
|
||||
|
||||
// Compress enabled content compression before upload.
|
||||
// Compression will typically reduce memory and network usage,
|
||||
// Compression can safely be enabled with MinIO hosts.
|
||||
Compress bool
|
||||
|
||||
// SkipErrs if enabled will skip any errors while reading the
|
||||
// object content while creating the snowball archive
|
||||
SkipErrs bool
|
||||
}
|
||||
|
||||
// SnowballObject contains information about a single object to be added to the snowball.
|
||||
type SnowballObject struct {
|
||||
// Key is the destination key, including prefix.
|
||||
Key string
|
||||
|
||||
// Size is the content size of this object.
|
||||
Size int64
|
||||
|
||||
// Modtime to apply to the object.
|
||||
// If Modtime is the zero value current time will be used.
|
||||
ModTime time.Time
|
||||
|
||||
// Content of the object.
|
||||
// Exactly 'Size' number of bytes must be provided.
|
||||
Content io.Reader
|
||||
|
||||
// VersionID of the object; if empty, a new versionID will be generated
|
||||
VersionID string
|
||||
|
||||
// Headers contains more options for this object upload, the same as you
|
||||
// would include in a regular PutObject operation, such as user metadata
|
||||
// and content-disposition, expires, ..
|
||||
Headers http.Header
|
||||
|
||||
// Close will be called when an object has finished processing.
|
||||
// Note that if PutObjectsSnowball returns because of an error,
|
||||
// objects not consumed from the input will NOT have been closed.
|
||||
// Leave as nil for no callback.
|
||||
Close func()
|
||||
}
|
||||
|
||||
type nopReadSeekCloser struct {
|
||||
io.ReadSeeker
|
||||
}
|
||||
|
||||
func (n nopReadSeekCloser) Close() error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// This is available as io.ReadSeekCloser from go1.16
|
||||
type readSeekCloser interface {
|
||||
io.Reader
|
||||
io.Closer
|
||||
io.Seeker
|
||||
}
|
||||
|
||||
// PutObjectsSnowball will put multiple objects with a single put call.
|
||||
// A (compressed) TAR file will be created which will contain multiple objects.
|
||||
// The key for each object will be used for the destination in the specified bucket.
|
||||
// Total size should be < 5TB.
|
||||
// This function blocks until 'objs' is closed and the content has been uploaded.
|
||||
func (c Client) PutObjectsSnowball(ctx context.Context, bucketName string, opts SnowballOptions, objs <-chan SnowballObject) (err error) {
|
||||
err = opts.Opts.validate(&c)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var tmpWriter io.Writer
|
||||
var getTmpReader func() (rc readSeekCloser, sz int64, err error)
|
||||
if opts.InMemory {
|
||||
b := bytes.NewBuffer(nil)
|
||||
tmpWriter = b
|
||||
getTmpReader = func() (readSeekCloser, int64, error) {
|
||||
return nopReadSeekCloser{bytes.NewReader(b.Bytes())}, int64(b.Len()), nil
|
||||
}
|
||||
} else {
|
||||
f, err := os.CreateTemp("", "s3-putsnowballobjects-*")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
name := f.Name()
|
||||
tmpWriter = f
|
||||
var once sync.Once
|
||||
defer once.Do(func() {
|
||||
f.Close()
|
||||
})
|
||||
defer os.Remove(name)
|
||||
getTmpReader = func() (readSeekCloser, int64, error) {
|
||||
once.Do(func() {
|
||||
f.Close()
|
||||
})
|
||||
f, err := os.Open(name)
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
st, err := f.Stat()
|
||||
if err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
return f, st.Size(), nil
|
||||
}
|
||||
}
|
||||
flush := func() error { return nil }
|
||||
if !opts.Compress {
|
||||
if !opts.InMemory {
|
||||
// Insert buffer for writes.
|
||||
buf := bufio.NewWriterSize(tmpWriter, 1<<20)
|
||||
flush = buf.Flush
|
||||
tmpWriter = buf
|
||||
}
|
||||
} else {
|
||||
s2c := s2.NewWriter(tmpWriter, s2.WriterBetterCompression())
|
||||
flush = s2c.Close
|
||||
defer s2c.Close()
|
||||
tmpWriter = s2c
|
||||
}
|
||||
t := tar.NewWriter(tmpWriter)
|
||||
|
||||
objectLoop:
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case obj, ok := <-objs:
|
||||
if !ok {
|
||||
break objectLoop
|
||||
}
|
||||
|
||||
closeObj := func() {}
|
||||
if obj.Close != nil {
|
||||
closeObj = obj.Close
|
||||
}
|
||||
|
||||
// Trim accidental slash prefix.
|
||||
obj.Key = strings.TrimPrefix(obj.Key, "/")
|
||||
header := tar.Header{
|
||||
Typeflag: tar.TypeReg,
|
||||
Name: obj.Key,
|
||||
Size: obj.Size,
|
||||
ModTime: obj.ModTime,
|
||||
Format: tar.FormatPAX,
|
||||
}
|
||||
if header.ModTime.IsZero() {
|
||||
header.ModTime = time.Now().UTC()
|
||||
}
|
||||
|
||||
header.PAXRecords = make(map[string]string)
|
||||
if obj.VersionID != "" {
|
||||
header.PAXRecords["minio.versionId"] = obj.VersionID
|
||||
}
|
||||
for k, vals := range obj.Headers {
|
||||
header.PAXRecords["minio.metadata."+k] = strings.Join(vals, ",")
|
||||
}
|
||||
|
||||
if err := t.WriteHeader(&header); err != nil {
|
||||
closeObj()
|
||||
return err
|
||||
}
|
||||
n, err := io.Copy(t, obj.Content)
|
||||
if err != nil {
|
||||
closeObj()
|
||||
if opts.SkipErrs {
|
||||
continue
|
||||
}
|
||||
return err
|
||||
}
|
||||
if n != obj.Size {
|
||||
closeObj()
|
||||
if opts.SkipErrs {
|
||||
continue
|
||||
}
|
||||
return io.ErrUnexpectedEOF
|
||||
}
|
||||
closeObj()
|
||||
}
|
||||
}
|
||||
// Flush tar
|
||||
err = t.Flush()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// Flush compression
|
||||
err = flush()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if opts.Opts.UserMetadata == nil {
|
||||
opts.Opts.UserMetadata = map[string]string{}
|
||||
}
|
||||
opts.Opts.UserMetadata["X-Amz-Meta-Snowball-Auto-Extract"] = "true"
|
||||
opts.Opts.DisableMultipart = true
|
||||
rc, sz, err := getTmpReader()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer rc.Close()
|
||||
rand := c.random.Uint64()
|
||||
_, err = c.PutObject(ctx, bucketName, fmt.Sprintf("snowball-upload-%x.tar", rand), rc, sz, opts.Opts)
|
||||
return err
|
||||
}
|
||||
+548
@@ -0,0 +1,548 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2015-2020 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/xml"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"time"
|
||||
|
||||
"github.com/minio/minio-go/v7/pkg/s3utils"
|
||||
)
|
||||
|
||||
//revive:disable
|
||||
|
||||
// Deprecated: BucketOptions will be renamed to RemoveBucketOptions in future versions.
|
||||
type BucketOptions = RemoveBucketOptions
|
||||
|
||||
//revive:enable
|
||||
|
||||
// RemoveBucketOptions special headers to purge buckets, only
|
||||
// useful when endpoint is MinIO
|
||||
type RemoveBucketOptions struct {
|
||||
ForceDelete bool
|
||||
}
|
||||
|
||||
// RemoveBucketWithOptions deletes the bucket name.
|
||||
//
|
||||
// All objects (including all object versions and delete markers)
|
||||
// in the bucket will be deleted forcibly if bucket options set
|
||||
// ForceDelete to 'true'.
|
||||
func (c *Client) RemoveBucketWithOptions(ctx context.Context, bucketName string, opts RemoveBucketOptions) error {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Build headers.
|
||||
headers := make(http.Header)
|
||||
if opts.ForceDelete {
|
||||
headers.Set(minIOForceDelete, "true")
|
||||
}
|
||||
|
||||
// Execute DELETE on bucket.
|
||||
resp, err := c.executeMethod(ctx, http.MethodDelete, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
contentSHA256Hex: emptySHA256Hex,
|
||||
customHeader: headers,
|
||||
})
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if resp != nil {
|
||||
if resp.StatusCode != http.StatusNoContent {
|
||||
return httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
}
|
||||
|
||||
// Remove the location from cache on a successful delete.
|
||||
c.bucketLocCache.Delete(bucketName)
|
||||
return nil
|
||||
}
|
||||
|
||||
// RemoveBucket deletes the bucket name.
|
||||
//
|
||||
// All objects (including all object versions and delete markers).
|
||||
// in the bucket must be deleted before successfully attempting this request.
|
||||
func (c *Client) RemoveBucket(ctx context.Context, bucketName string) error {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return err
|
||||
}
|
||||
// Execute DELETE on bucket.
|
||||
resp, err := c.executeMethod(ctx, http.MethodDelete, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
contentSHA256Hex: emptySHA256Hex,
|
||||
})
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if resp != nil {
|
||||
if resp.StatusCode != http.StatusNoContent {
|
||||
return httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
}
|
||||
|
||||
// Remove the location from cache on a successful delete.
|
||||
c.bucketLocCache.Delete(bucketName)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// AdvancedRemoveOptions intended for internal use by replication
|
||||
type AdvancedRemoveOptions struct {
|
||||
ReplicationDeleteMarker bool
|
||||
ReplicationStatus ReplicationStatus
|
||||
ReplicationMTime time.Time
|
||||
ReplicationRequest bool
|
||||
ReplicationValidityCheck bool // check permissions
|
||||
}
|
||||
|
||||
// RemoveObjectOptions represents options specified by user for RemoveObject call
|
||||
type RemoveObjectOptions struct {
|
||||
ForceDelete bool
|
||||
GovernanceBypass bool
|
||||
VersionID string
|
||||
Internal AdvancedRemoveOptions
|
||||
}
|
||||
|
||||
// RemoveObject removes an object from a bucket.
|
||||
func (c *Client) RemoveObject(ctx context.Context, bucketName, objectName string, opts RemoveObjectOptions) error {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := s3utils.CheckValidObjectName(objectName); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
res := c.removeObject(ctx, bucketName, objectName, opts)
|
||||
return res.Err
|
||||
}
|
||||
|
||||
func (c *Client) removeObject(ctx context.Context, bucketName, objectName string, opts RemoveObjectOptions) RemoveObjectResult {
|
||||
// Get resources properly escaped and lined up before
|
||||
// using them in http request.
|
||||
urlValues := make(url.Values)
|
||||
|
||||
if opts.VersionID != "" {
|
||||
urlValues.Set("versionId", opts.VersionID)
|
||||
}
|
||||
|
||||
// Build headers.
|
||||
headers := make(http.Header)
|
||||
|
||||
if opts.GovernanceBypass {
|
||||
// Set the bypass goverenance retention header
|
||||
headers.Set(amzBypassGovernance, "true")
|
||||
}
|
||||
if opts.Internal.ReplicationDeleteMarker {
|
||||
headers.Set(minIOBucketReplicationDeleteMarker, "true")
|
||||
}
|
||||
if !opts.Internal.ReplicationMTime.IsZero() {
|
||||
headers.Set(minIOBucketSourceMTime, opts.Internal.ReplicationMTime.Format(time.RFC3339Nano))
|
||||
}
|
||||
if !opts.Internal.ReplicationStatus.Empty() {
|
||||
headers.Set(amzBucketReplicationStatus, string(opts.Internal.ReplicationStatus))
|
||||
}
|
||||
if opts.Internal.ReplicationRequest {
|
||||
headers.Set(minIOBucketReplicationRequest, "true")
|
||||
}
|
||||
if opts.Internal.ReplicationValidityCheck {
|
||||
headers.Set(minIOBucketReplicationCheck, "true")
|
||||
}
|
||||
if opts.ForceDelete {
|
||||
headers.Set(minIOForceDelete, "true")
|
||||
}
|
||||
// Execute DELETE on objectName.
|
||||
resp, err := c.executeMethod(ctx, http.MethodDelete, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
objectName: objectName,
|
||||
contentSHA256Hex: emptySHA256Hex,
|
||||
queryValues: urlValues,
|
||||
customHeader: headers,
|
||||
})
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return RemoveObjectResult{Err: err}
|
||||
}
|
||||
if resp != nil {
|
||||
// if some unexpected error happened and max retry is reached, we want to let client know
|
||||
if resp.StatusCode != http.StatusNoContent {
|
||||
err := httpRespToErrorResponse(resp, bucketName, objectName)
|
||||
return RemoveObjectResult{Err: err}
|
||||
}
|
||||
}
|
||||
|
||||
// DeleteObject always responds with http '204' even for
|
||||
// objects which do not exist. So no need to handle them
|
||||
// specifically.
|
||||
return RemoveObjectResult{
|
||||
ObjectName: objectName,
|
||||
ObjectVersionID: opts.VersionID,
|
||||
DeleteMarker: resp.Header.Get("x-amz-delete-marker") == "true",
|
||||
DeleteMarkerVersionID: resp.Header.Get("x-amz-version-id"),
|
||||
}
|
||||
}
|
||||
|
||||
// RemoveObjectError - container of Multi Delete S3 API error
|
||||
type RemoveObjectError struct {
|
||||
ObjectName string
|
||||
VersionID string
|
||||
Err error
|
||||
}
|
||||
|
||||
// RemoveObjectResult - container of Multi Delete S3 API result
|
||||
type RemoveObjectResult struct {
|
||||
ObjectName string
|
||||
ObjectVersionID string
|
||||
|
||||
DeleteMarker bool
|
||||
DeleteMarkerVersionID string
|
||||
|
||||
Err error
|
||||
}
|
||||
|
||||
// generateRemoveMultiObjects - generate the XML request for remove multi objects request
|
||||
func generateRemoveMultiObjectsRequest(objects []ObjectInfo) []byte {
|
||||
delObjects := []deleteObject{}
|
||||
for _, obj := range objects {
|
||||
delObjects = append(delObjects, deleteObject{
|
||||
Key: obj.Key,
|
||||
VersionID: obj.VersionID,
|
||||
})
|
||||
}
|
||||
xmlBytes, _ := xml.Marshal(deleteMultiObjects{Objects: delObjects, Quiet: false})
|
||||
return xmlBytes
|
||||
}
|
||||
|
||||
// processRemoveMultiObjectsResponse - parse the remove multi objects web service
|
||||
// and return the success/failure result status for each object
|
||||
func processRemoveMultiObjectsResponse(body io.Reader, resultCh chan<- RemoveObjectResult) {
|
||||
// Parse multi delete XML response
|
||||
rmResult := &deleteMultiObjectsResult{}
|
||||
err := xmlDecoder(body, rmResult)
|
||||
if err != nil {
|
||||
resultCh <- RemoveObjectResult{ObjectName: "", Err: err}
|
||||
return
|
||||
}
|
||||
|
||||
// Fill deletion that returned success
|
||||
for _, obj := range rmResult.DeletedObjects {
|
||||
resultCh <- RemoveObjectResult{
|
||||
ObjectName: obj.Key,
|
||||
// Only filled with versioned buckets
|
||||
ObjectVersionID: obj.VersionID,
|
||||
DeleteMarker: obj.DeleteMarker,
|
||||
DeleteMarkerVersionID: obj.DeleteMarkerVersionID,
|
||||
}
|
||||
}
|
||||
|
||||
// Fill deletion that returned an error.
|
||||
for _, obj := range rmResult.UnDeletedObjects {
|
||||
// Version does not exist is not an error ignore and continue.
|
||||
switch obj.Code {
|
||||
case "InvalidArgument", "NoSuchVersion":
|
||||
continue
|
||||
}
|
||||
resultCh <- RemoveObjectResult{
|
||||
ObjectName: obj.Key,
|
||||
ObjectVersionID: obj.VersionID,
|
||||
Err: ErrorResponse{
|
||||
Code: obj.Code,
|
||||
Message: obj.Message,
|
||||
},
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// RemoveObjectsOptions represents options specified by user for RemoveObjects call
|
||||
type RemoveObjectsOptions struct {
|
||||
GovernanceBypass bool
|
||||
}
|
||||
|
||||
// RemoveObjects removes multiple objects from a bucket while
|
||||
// it is possible to specify objects versions which are received from
|
||||
// objectsCh. Remove failures are sent back via error channel.
|
||||
func (c *Client) RemoveObjects(ctx context.Context, bucketName string, objectsCh <-chan ObjectInfo, opts RemoveObjectsOptions) <-chan RemoveObjectError {
|
||||
errorCh := make(chan RemoveObjectError, 1)
|
||||
|
||||
// Validate if bucket name is valid.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
defer close(errorCh)
|
||||
errorCh <- RemoveObjectError{
|
||||
Err: err,
|
||||
}
|
||||
return errorCh
|
||||
}
|
||||
// Validate objects channel to be properly allocated.
|
||||
if objectsCh == nil {
|
||||
defer close(errorCh)
|
||||
errorCh <- RemoveObjectError{
|
||||
Err: errInvalidArgument("Objects channel cannot be nil"),
|
||||
}
|
||||
return errorCh
|
||||
}
|
||||
|
||||
resultCh := make(chan RemoveObjectResult, 1)
|
||||
go c.removeObjects(ctx, bucketName, objectsCh, resultCh, opts)
|
||||
go func() {
|
||||
defer close(errorCh)
|
||||
for res := range resultCh {
|
||||
// Send only errors to the error channel
|
||||
if res.Err == nil {
|
||||
continue
|
||||
}
|
||||
errorCh <- RemoveObjectError{
|
||||
ObjectName: res.ObjectName,
|
||||
VersionID: res.ObjectVersionID,
|
||||
Err: res.Err,
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
return errorCh
|
||||
}
|
||||
|
||||
// RemoveObjectsWithResult removes multiple objects from a bucket while
|
||||
// it is possible to specify objects versions which are received from
|
||||
// objectsCh. Remove results, successes and failures are sent back via
|
||||
// RemoveObjectResult channel
|
||||
func (c *Client) RemoveObjectsWithResult(ctx context.Context, bucketName string, objectsCh <-chan ObjectInfo, opts RemoveObjectsOptions) <-chan RemoveObjectResult {
|
||||
resultCh := make(chan RemoveObjectResult, 1)
|
||||
|
||||
// Validate if bucket name is valid.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
defer close(resultCh)
|
||||
resultCh <- RemoveObjectResult{
|
||||
Err: err,
|
||||
}
|
||||
return resultCh
|
||||
}
|
||||
// Validate objects channel to be properly allocated.
|
||||
if objectsCh == nil {
|
||||
defer close(resultCh)
|
||||
resultCh <- RemoveObjectResult{
|
||||
Err: errInvalidArgument("Objects channel cannot be nil"),
|
||||
}
|
||||
return resultCh
|
||||
}
|
||||
|
||||
go c.removeObjects(ctx, bucketName, objectsCh, resultCh, opts)
|
||||
return resultCh
|
||||
}
|
||||
|
||||
// Return true if the character is within the allowed characters in an XML 1.0 document
|
||||
// The list of allowed characters can be found here: https://www.w3.org/TR/xml/#charsets
|
||||
func validXMLChar(r rune) (ok bool) {
|
||||
return r == 0x09 ||
|
||||
r == 0x0A ||
|
||||
r == 0x0D ||
|
||||
r >= 0x20 && r <= 0xD7FF ||
|
||||
r >= 0xE000 && r <= 0xFFFD ||
|
||||
r >= 0x10000 && r <= 0x10FFFF
|
||||
}
|
||||
|
||||
func hasInvalidXMLChar(str string) bool {
|
||||
for _, s := range str {
|
||||
if !validXMLChar(s) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Generate and call MultiDelete S3 requests based on entries received from objectsCh
|
||||
func (c *Client) removeObjects(ctx context.Context, bucketName string, objectsCh <-chan ObjectInfo, resultCh chan<- RemoveObjectResult, opts RemoveObjectsOptions) {
|
||||
maxEntries := 1000
|
||||
finish := false
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("delete", "")
|
||||
|
||||
// Close result channel when Multi delete finishes.
|
||||
defer close(resultCh)
|
||||
|
||||
// Loop over entries by 1000 and call MultiDelete requests
|
||||
for {
|
||||
if finish {
|
||||
break
|
||||
}
|
||||
count := 0
|
||||
var batch []ObjectInfo
|
||||
|
||||
// Try to gather 1000 entries
|
||||
for object := range objectsCh {
|
||||
if hasInvalidXMLChar(object.Key) {
|
||||
// Use single DELETE so the object name will be in the request URL instead of the multi-delete XML document.
|
||||
removeResult := c.removeObject(ctx, bucketName, object.Key, RemoveObjectOptions{
|
||||
VersionID: object.VersionID,
|
||||
GovernanceBypass: opts.GovernanceBypass,
|
||||
})
|
||||
if err := removeResult.Err; err != nil {
|
||||
// Version does not exist is not an error ignore and continue.
|
||||
switch ToErrorResponse(err).Code {
|
||||
case "InvalidArgument", "NoSuchVersion":
|
||||
continue
|
||||
}
|
||||
resultCh <- removeResult
|
||||
}
|
||||
|
||||
resultCh <- removeResult
|
||||
continue
|
||||
}
|
||||
|
||||
batch = append(batch, object)
|
||||
if count++; count >= maxEntries {
|
||||
break
|
||||
}
|
||||
}
|
||||
if count == 0 {
|
||||
// Multi Objects Delete API doesn't accept empty object list, quit immediately
|
||||
break
|
||||
}
|
||||
if count < maxEntries {
|
||||
// We didn't have 1000 entries, so this is the last batch
|
||||
finish = true
|
||||
}
|
||||
|
||||
// Build headers.
|
||||
headers := make(http.Header)
|
||||
if opts.GovernanceBypass {
|
||||
// Set the bypass goverenance retention header
|
||||
headers.Set(amzBypassGovernance, "true")
|
||||
}
|
||||
|
||||
// Generate remove multi objects XML request
|
||||
removeBytes := generateRemoveMultiObjectsRequest(batch)
|
||||
// Execute POST on bucket to remove objects.
|
||||
resp, err := c.executeMethod(ctx, http.MethodPost, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
queryValues: urlValues,
|
||||
contentBody: bytes.NewReader(removeBytes),
|
||||
contentLength: int64(len(removeBytes)),
|
||||
contentMD5Base64: sumMD5Base64(removeBytes),
|
||||
contentSHA256Hex: sum256Hex(removeBytes),
|
||||
customHeader: headers,
|
||||
})
|
||||
if resp != nil {
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
e := httpRespToErrorResponse(resp, bucketName, "")
|
||||
resultCh <- RemoveObjectResult{ObjectName: "", Err: e}
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
for _, b := range batch {
|
||||
resultCh <- RemoveObjectResult{
|
||||
ObjectName: b.Key,
|
||||
ObjectVersionID: b.VersionID,
|
||||
Err: err,
|
||||
}
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
// Process multiobjects remove xml response
|
||||
processRemoveMultiObjectsResponse(resp.Body, resultCh)
|
||||
|
||||
closeResponse(resp)
|
||||
}
|
||||
}
|
||||
|
||||
// RemoveIncompleteUpload aborts an partially uploaded object.
|
||||
func (c *Client) RemoveIncompleteUpload(ctx context.Context, bucketName, objectName string) error {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := s3utils.CheckValidObjectName(objectName); err != nil {
|
||||
return err
|
||||
}
|
||||
// Find multipart upload ids of the object to be aborted.
|
||||
uploadIDs, err := c.findUploadIDs(ctx, bucketName, objectName)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for _, uploadID := range uploadIDs {
|
||||
// abort incomplete multipart upload, based on the upload id passed.
|
||||
err := c.abortMultipartUpload(ctx, bucketName, objectName, uploadID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// abortMultipartUpload aborts a multipart upload for the given
|
||||
// uploadID, all previously uploaded parts are deleted.
|
||||
func (c *Client) abortMultipartUpload(ctx context.Context, bucketName, objectName, uploadID string) error {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := s3utils.CheckValidObjectName(objectName); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Initialize url queries.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("uploadId", uploadID)
|
||||
|
||||
// Execute DELETE on multipart upload.
|
||||
resp, err := c.executeMethod(ctx, http.MethodDelete, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
objectName: objectName,
|
||||
queryValues: urlValues,
|
||||
contentSHA256Hex: emptySHA256Hex,
|
||||
})
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if resp != nil {
|
||||
if resp.StatusCode != http.StatusNoContent {
|
||||
// Abort has no response body, handle it for any errors.
|
||||
var errorResponse ErrorResponse
|
||||
switch resp.StatusCode {
|
||||
case http.StatusNotFound:
|
||||
// This is needed specifically for abort and it cannot
|
||||
// be converged into default case.
|
||||
errorResponse = ErrorResponse{
|
||||
Code: "NoSuchUpload",
|
||||
Message: "The specified multipart upload does not exist.",
|
||||
BucketName: bucketName,
|
||||
Key: objectName,
|
||||
RequestID: resp.Header.Get("x-amz-request-id"),
|
||||
HostID: resp.Header.Get("x-amz-id-2"),
|
||||
Region: resp.Header.Get("x-amz-bucket-region"),
|
||||
}
|
||||
default:
|
||||
return httpRespToErrorResponse(resp, bucketName, objectName)
|
||||
}
|
||||
return errorResponse
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
+182
@@ -0,0 +1,182 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* (C) 2018-2021 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/xml"
|
||||
"net/http"
|
||||
"net/url"
|
||||
|
||||
"github.com/minio/minio-go/v7/pkg/s3utils"
|
||||
"github.com/minio/minio-go/v7/pkg/tags"
|
||||
)
|
||||
|
||||
// RestoreType represents the restore request type
|
||||
type RestoreType string
|
||||
|
||||
const (
|
||||
// RestoreSelect represents the restore SELECT operation
|
||||
RestoreSelect = RestoreType("SELECT")
|
||||
)
|
||||
|
||||
// TierType represents a retrieval tier
|
||||
type TierType string
|
||||
|
||||
const (
|
||||
// TierStandard is the standard retrieval tier
|
||||
TierStandard = TierType("Standard")
|
||||
// TierBulk is the bulk retrieval tier
|
||||
TierBulk = TierType("Bulk")
|
||||
// TierExpedited is the expedited retrieval tier
|
||||
TierExpedited = TierType("Expedited")
|
||||
)
|
||||
|
||||
// GlacierJobParameters represents the retrieval tier parameter
|
||||
type GlacierJobParameters struct {
|
||||
Tier TierType
|
||||
}
|
||||
|
||||
// Encryption contains the type of server-side encryption used during object retrieval
|
||||
type Encryption struct {
|
||||
EncryptionType string
|
||||
KMSContext string
|
||||
KMSKeyID string `xml:"KMSKeyId"`
|
||||
}
|
||||
|
||||
// MetadataEntry represents a metadata information of the restored object.
|
||||
type MetadataEntry struct {
|
||||
Name string
|
||||
Value string
|
||||
}
|
||||
|
||||
// S3 holds properties of the copy of the archived object
|
||||
type S3 struct {
|
||||
AccessControlList *AccessControlList `xml:"AccessControlList,omitempty"`
|
||||
BucketName string
|
||||
Prefix string
|
||||
CannedACL *string `xml:"CannedACL,omitempty"`
|
||||
Encryption *Encryption `xml:"Encryption,omitempty"`
|
||||
StorageClass *string `xml:"StorageClass,omitempty"`
|
||||
Tagging *tags.Tags `xml:"Tagging,omitempty"`
|
||||
UserMetadata *MetadataEntry `xml:"UserMetadata,omitempty"`
|
||||
}
|
||||
|
||||
// SelectParameters holds the select request parameters
|
||||
type SelectParameters struct {
|
||||
XMLName xml.Name `xml:"SelectParameters"`
|
||||
ExpressionType QueryExpressionType
|
||||
Expression string
|
||||
InputSerialization SelectObjectInputSerialization
|
||||
OutputSerialization SelectObjectOutputSerialization
|
||||
}
|
||||
|
||||
// OutputLocation holds properties of the copy of the archived object
|
||||
type OutputLocation struct {
|
||||
XMLName xml.Name `xml:"OutputLocation"`
|
||||
S3 S3 `xml:"S3"`
|
||||
}
|
||||
|
||||
// RestoreRequest holds properties of the restore object request
|
||||
type RestoreRequest struct {
|
||||
XMLName xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ RestoreRequest"`
|
||||
Type *RestoreType `xml:"Type,omitempty"`
|
||||
Tier *TierType `xml:"Tier,omitempty"`
|
||||
Days *int `xml:"Days,omitempty"`
|
||||
GlacierJobParameters *GlacierJobParameters `xml:"GlacierJobParameters,omitempty"`
|
||||
Description *string `xml:"Description,omitempty"`
|
||||
SelectParameters *SelectParameters `xml:"SelectParameters,omitempty"`
|
||||
OutputLocation *OutputLocation `xml:"OutputLocation,omitempty"`
|
||||
}
|
||||
|
||||
// SetDays sets the days parameter of the restore request
|
||||
func (r *RestoreRequest) SetDays(v int) {
|
||||
r.Days = &v
|
||||
}
|
||||
|
||||
// SetGlacierJobParameters sets the GlacierJobParameters of the restore request
|
||||
func (r *RestoreRequest) SetGlacierJobParameters(v GlacierJobParameters) {
|
||||
r.GlacierJobParameters = &v
|
||||
}
|
||||
|
||||
// SetType sets the type of the restore request
|
||||
func (r *RestoreRequest) SetType(v RestoreType) {
|
||||
r.Type = &v
|
||||
}
|
||||
|
||||
// SetTier sets the retrieval tier of the restore request
|
||||
func (r *RestoreRequest) SetTier(v TierType) {
|
||||
r.Tier = &v
|
||||
}
|
||||
|
||||
// SetDescription sets the description of the restore request
|
||||
func (r *RestoreRequest) SetDescription(v string) {
|
||||
r.Description = &v
|
||||
}
|
||||
|
||||
// SetSelectParameters sets SelectParameters of the restore select request
|
||||
func (r *RestoreRequest) SetSelectParameters(v SelectParameters) {
|
||||
r.SelectParameters = &v
|
||||
}
|
||||
|
||||
// SetOutputLocation sets the properties of the copy of the archived object
|
||||
func (r *RestoreRequest) SetOutputLocation(v OutputLocation) {
|
||||
r.OutputLocation = &v
|
||||
}
|
||||
|
||||
// RestoreObject is a implementation of https://docs.aws.amazon.com/AmazonS3/latest/API/API_RestoreObject.html AWS S3 API
|
||||
func (c *Client) RestoreObject(ctx context.Context, bucketName, objectName, versionID string, req RestoreRequest) error {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := s3utils.CheckValidObjectName(objectName); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
restoreRequestBytes, err := xml.Marshal(req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("restore", "")
|
||||
if versionID != "" {
|
||||
urlValues.Set("versionId", versionID)
|
||||
}
|
||||
|
||||
// Execute POST on bucket/object.
|
||||
resp, err := c.executeMethod(ctx, http.MethodPost, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
objectName: objectName,
|
||||
queryValues: urlValues,
|
||||
contentMD5Base64: sumMD5Base64(restoreRequestBytes),
|
||||
contentSHA256Hex: sum256Hex(restoreRequestBytes),
|
||||
contentBody: bytes.NewReader(restoreRequestBytes),
|
||||
contentLength: int64(len(restoreRequestBytes)),
|
||||
})
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if resp.StatusCode != http.StatusAccepted && resp.StatusCode != http.StatusOK {
|
||||
return httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
+452
@@ -0,0 +1,452 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2015-2020 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"encoding/base64"
|
||||
"encoding/xml"
|
||||
"errors"
|
||||
"io"
|
||||
"reflect"
|
||||
"time"
|
||||
)
|
||||
|
||||
// listAllMyBucketsResult container for listBuckets response.
|
||||
type listAllMyBucketsResult struct {
|
||||
// Container for one or more buckets.
|
||||
Buckets struct {
|
||||
Bucket []BucketInfo
|
||||
}
|
||||
Owner owner
|
||||
}
|
||||
|
||||
// owner container for bucket owner information.
|
||||
type owner struct {
|
||||
DisplayName string
|
||||
ID string
|
||||
}
|
||||
|
||||
// CommonPrefix container for prefix response.
|
||||
type CommonPrefix struct {
|
||||
Prefix string
|
||||
}
|
||||
|
||||
// ListBucketV2Result container for listObjects response version 2.
|
||||
type ListBucketV2Result struct {
|
||||
// A response can contain CommonPrefixes only if you have
|
||||
// specified a delimiter.
|
||||
CommonPrefixes []CommonPrefix
|
||||
// Metadata about each object returned.
|
||||
Contents []ObjectInfo
|
||||
Delimiter string
|
||||
|
||||
// Encoding type used to encode object keys in the response.
|
||||
EncodingType string
|
||||
|
||||
// A flag that indicates whether or not ListObjects returned all of the results
|
||||
// that satisfied the search criteria.
|
||||
IsTruncated bool
|
||||
MaxKeys int64
|
||||
Name string
|
||||
|
||||
// Hold the token that will be sent in the next request to fetch the next group of keys
|
||||
NextContinuationToken string
|
||||
|
||||
ContinuationToken string
|
||||
Prefix string
|
||||
|
||||
// FetchOwner and StartAfter are currently not used
|
||||
FetchOwner string
|
||||
StartAfter string
|
||||
}
|
||||
|
||||
// Version is an element in the list object versions response
|
||||
type Version struct {
|
||||
ETag string
|
||||
IsLatest bool
|
||||
Key string
|
||||
LastModified time.Time
|
||||
Owner Owner
|
||||
Size int64
|
||||
StorageClass string
|
||||
VersionID string `xml:"VersionId"`
|
||||
|
||||
// x-amz-meta-* headers stripped "x-amz-meta-" prefix containing the first value.
|
||||
// Only returned by MinIO servers.
|
||||
UserMetadata StringMap `json:"userMetadata,omitempty"`
|
||||
|
||||
// x-amz-tagging values in their k/v values.
|
||||
// Only returned by MinIO servers.
|
||||
UserTags URLMap `json:"userTags,omitempty" xml:"UserTags"`
|
||||
|
||||
Internal *struct {
|
||||
K int // Data blocks
|
||||
M int // Parity blocks
|
||||
} `xml:"Internal"`
|
||||
|
||||
isDeleteMarker bool
|
||||
}
|
||||
|
||||
// ListVersionsResult is an element in the list object versions response
|
||||
// and has a special Unmarshaler because we need to preserver the order
|
||||
// of <Version> and <DeleteMarker> in ListVersionsResult.Versions slice
|
||||
type ListVersionsResult struct {
|
||||
Versions []Version
|
||||
|
||||
CommonPrefixes []CommonPrefix
|
||||
Name string
|
||||
Prefix string
|
||||
Delimiter string
|
||||
MaxKeys int64
|
||||
EncodingType string
|
||||
IsTruncated bool
|
||||
KeyMarker string
|
||||
VersionIDMarker string
|
||||
NextKeyMarker string
|
||||
NextVersionIDMarker string
|
||||
}
|
||||
|
||||
// UnmarshalXML is a custom unmarshal code for the response of ListObjectVersions, the custom
|
||||
// code will unmarshal <Version> and <DeleteMarker> tags and save them in Versions field to
|
||||
// preserve the lexical order of the listing.
|
||||
func (l *ListVersionsResult) UnmarshalXML(d *xml.Decoder, _ xml.StartElement) (err error) {
|
||||
for {
|
||||
// Read tokens from the XML document in a stream.
|
||||
t, err := d.Token()
|
||||
if err != nil {
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
se, ok := t.(xml.StartElement)
|
||||
if ok {
|
||||
tagName := se.Name.Local
|
||||
switch tagName {
|
||||
case "Name", "Prefix",
|
||||
"Delimiter", "EncodingType",
|
||||
"KeyMarker", "NextKeyMarker":
|
||||
var s string
|
||||
if err = d.DecodeElement(&s, &se); err != nil {
|
||||
return err
|
||||
}
|
||||
v := reflect.ValueOf(l).Elem().FieldByName(tagName)
|
||||
if v.IsValid() {
|
||||
v.SetString(s)
|
||||
}
|
||||
case "VersionIdMarker":
|
||||
// VersionIdMarker is a special case because of 'Id' instead of 'ID' in field name
|
||||
var s string
|
||||
if err = d.DecodeElement(&s, &se); err != nil {
|
||||
return err
|
||||
}
|
||||
l.VersionIDMarker = s
|
||||
case "NextVersionIdMarker":
|
||||
// NextVersionIdMarker is a special case because of 'Id' instead of 'ID' in field name
|
||||
var s string
|
||||
if err = d.DecodeElement(&s, &se); err != nil {
|
||||
return err
|
||||
}
|
||||
l.NextVersionIDMarker = s
|
||||
case "IsTruncated": // bool
|
||||
var b bool
|
||||
if err = d.DecodeElement(&b, &se); err != nil {
|
||||
return err
|
||||
}
|
||||
l.IsTruncated = b
|
||||
case "MaxKeys": // int64
|
||||
var i int64
|
||||
if err = d.DecodeElement(&i, &se); err != nil {
|
||||
return err
|
||||
}
|
||||
l.MaxKeys = i
|
||||
case "CommonPrefixes":
|
||||
var cp CommonPrefix
|
||||
if err = d.DecodeElement(&cp, &se); err != nil {
|
||||
return err
|
||||
}
|
||||
l.CommonPrefixes = append(l.CommonPrefixes, cp)
|
||||
case "DeleteMarker", "Version":
|
||||
var v Version
|
||||
if err = d.DecodeElement(&v, &se); err != nil {
|
||||
return err
|
||||
}
|
||||
if tagName == "DeleteMarker" {
|
||||
v.isDeleteMarker = true
|
||||
}
|
||||
l.Versions = append(l.Versions, v)
|
||||
default:
|
||||
return errors.New("unrecognized option:" + tagName)
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ListBucketResult container for listObjects response.
|
||||
type ListBucketResult struct {
|
||||
// A response can contain CommonPrefixes only if you have
|
||||
// specified a delimiter.
|
||||
CommonPrefixes []CommonPrefix
|
||||
// Metadata about each object returned.
|
||||
Contents []ObjectInfo
|
||||
Delimiter string
|
||||
|
||||
// Encoding type used to encode object keys in the response.
|
||||
EncodingType string
|
||||
|
||||
// A flag that indicates whether or not ListObjects returned all of the results
|
||||
// that satisfied the search criteria.
|
||||
IsTruncated bool
|
||||
Marker string
|
||||
MaxKeys int64
|
||||
Name string
|
||||
|
||||
// When response is truncated (the IsTruncated element value in
|
||||
// the response is true), you can use the key name in this field
|
||||
// as marker in the subsequent request to get next set of objects.
|
||||
// Object storage lists objects in alphabetical order Note: This
|
||||
// element is returned only if you have delimiter request
|
||||
// parameter specified. If response does not include the NextMaker
|
||||
// and it is truncated, you can use the value of the last Key in
|
||||
// the response as the marker in the subsequent request to get the
|
||||
// next set of object keys.
|
||||
NextMarker string
|
||||
Prefix string
|
||||
}
|
||||
|
||||
// ListMultipartUploadsResult container for ListMultipartUploads response
|
||||
type ListMultipartUploadsResult struct {
|
||||
Bucket string
|
||||
KeyMarker string
|
||||
UploadIDMarker string `xml:"UploadIdMarker"`
|
||||
NextKeyMarker string
|
||||
NextUploadIDMarker string `xml:"NextUploadIdMarker"`
|
||||
EncodingType string
|
||||
MaxUploads int64
|
||||
IsTruncated bool
|
||||
Uploads []ObjectMultipartInfo `xml:"Upload"`
|
||||
Prefix string
|
||||
Delimiter string
|
||||
// A response can contain CommonPrefixes only if you specify a delimiter.
|
||||
CommonPrefixes []CommonPrefix
|
||||
}
|
||||
|
||||
// initiator container for who initiated multipart upload.
|
||||
type initiator struct {
|
||||
ID string
|
||||
DisplayName string
|
||||
}
|
||||
|
||||
// copyObjectResult container for copy object response.
|
||||
type copyObjectResult struct {
|
||||
ETag string
|
||||
LastModified time.Time // time string format "2006-01-02T15:04:05.000Z"
|
||||
}
|
||||
|
||||
// ObjectPart container for particular part of an object.
|
||||
type ObjectPart struct {
|
||||
// Part number identifies the part.
|
||||
PartNumber int
|
||||
|
||||
// Date and time the part was uploaded.
|
||||
LastModified time.Time
|
||||
|
||||
// Entity tag returned when the part was uploaded, usually md5sum
|
||||
// of the part.
|
||||
ETag string
|
||||
|
||||
// Size of the uploaded part data.
|
||||
Size int64
|
||||
|
||||
// Checksum values of each part.
|
||||
ChecksumCRC32 string
|
||||
ChecksumCRC32C string
|
||||
ChecksumSHA1 string
|
||||
ChecksumSHA256 string
|
||||
ChecksumCRC64NVME string
|
||||
}
|
||||
|
||||
// Checksum will return the checksum for the given type.
|
||||
// Will return the empty string if not set.
|
||||
func (c ObjectPart) Checksum(t ChecksumType) string {
|
||||
switch {
|
||||
case t.Is(ChecksumCRC32C):
|
||||
return c.ChecksumCRC32C
|
||||
case t.Is(ChecksumCRC32):
|
||||
return c.ChecksumCRC32
|
||||
case t.Is(ChecksumSHA1):
|
||||
return c.ChecksumSHA1
|
||||
case t.Is(ChecksumSHA256):
|
||||
return c.ChecksumSHA256
|
||||
case t.Is(ChecksumCRC64NVME):
|
||||
return c.ChecksumCRC64NVME
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// ChecksumRaw returns the decoded checksum from the part.
|
||||
func (c ObjectPart) ChecksumRaw(t ChecksumType) ([]byte, error) {
|
||||
b64 := c.Checksum(t)
|
||||
if b64 == "" {
|
||||
return nil, errors.New("no checksum set")
|
||||
}
|
||||
decoded, err := base64.StdEncoding.DecodeString(b64)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(decoded) != t.RawByteLen() {
|
||||
return nil, errors.New("checksum length mismatch")
|
||||
}
|
||||
return decoded, nil
|
||||
}
|
||||
|
||||
// ListObjectPartsResult container for ListObjectParts response.
|
||||
type ListObjectPartsResult struct {
|
||||
Bucket string
|
||||
Key string
|
||||
UploadID string `xml:"UploadId"`
|
||||
|
||||
Initiator initiator
|
||||
Owner owner
|
||||
|
||||
StorageClass string
|
||||
PartNumberMarker int
|
||||
NextPartNumberMarker int
|
||||
MaxParts int
|
||||
|
||||
// ChecksumAlgorithm will be CRC32, CRC32C, etc.
|
||||
ChecksumAlgorithm string
|
||||
|
||||
// ChecksumType is FULL_OBJECT or COMPOSITE (assume COMPOSITE when unset)
|
||||
ChecksumType string
|
||||
|
||||
// Indicates whether the returned list of parts is truncated.
|
||||
IsTruncated bool
|
||||
ObjectParts []ObjectPart `xml:"Part"`
|
||||
|
||||
EncodingType string
|
||||
}
|
||||
|
||||
// initiateMultipartUploadResult container for InitiateMultiPartUpload
|
||||
// response.
|
||||
type initiateMultipartUploadResult struct {
|
||||
Bucket string
|
||||
Key string
|
||||
UploadID string `xml:"UploadId"`
|
||||
}
|
||||
|
||||
// completeMultipartUploadResult container for completed multipart
|
||||
// upload response.
|
||||
type completeMultipartUploadResult struct {
|
||||
Location string
|
||||
Bucket string
|
||||
Key string
|
||||
ETag string
|
||||
|
||||
// Checksum values, hash of hashes of parts.
|
||||
ChecksumCRC32 string
|
||||
ChecksumCRC32C string
|
||||
ChecksumSHA1 string
|
||||
ChecksumSHA256 string
|
||||
ChecksumCRC64NVME string
|
||||
}
|
||||
|
||||
// CompletePart sub container lists individual part numbers and their
|
||||
// md5sum, part of completeMultipartUpload.
|
||||
type CompletePart struct {
|
||||
// Part number identifies the part.
|
||||
PartNumber int
|
||||
ETag string
|
||||
|
||||
// Checksum values
|
||||
ChecksumCRC32 string `xml:"ChecksumCRC32,omitempty"`
|
||||
ChecksumCRC32C string `xml:"ChecksumCRC32C,omitempty"`
|
||||
ChecksumSHA1 string `xml:"ChecksumSHA1,omitempty"`
|
||||
ChecksumSHA256 string `xml:"ChecksumSHA256,omitempty"`
|
||||
ChecksumCRC64NVME string `xml:",omitempty"`
|
||||
}
|
||||
|
||||
// Checksum will return the checksum for the given type.
|
||||
// Will return the empty string if not set.
|
||||
func (c CompletePart) Checksum(t ChecksumType) string {
|
||||
switch {
|
||||
case t.Is(ChecksumCRC32C):
|
||||
return c.ChecksumCRC32C
|
||||
case t.Is(ChecksumCRC32):
|
||||
return c.ChecksumCRC32
|
||||
case t.Is(ChecksumSHA1):
|
||||
return c.ChecksumSHA1
|
||||
case t.Is(ChecksumSHA256):
|
||||
return c.ChecksumSHA256
|
||||
case t.Is(ChecksumCRC64NVME):
|
||||
return c.ChecksumCRC64NVME
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// completeMultipartUpload container for completing multipart upload.
|
||||
type completeMultipartUpload struct {
|
||||
XMLName xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ CompleteMultipartUpload" json:"-"`
|
||||
Parts []CompletePart `xml:"Part"`
|
||||
}
|
||||
|
||||
// createBucketConfiguration container for bucket configuration.
|
||||
type createBucketConfiguration struct {
|
||||
XMLName xml.Name `xml:"http://s3.amazonaws.com/doc/2006-03-01/ CreateBucketConfiguration" json:"-"`
|
||||
Location string `xml:"LocationConstraint"`
|
||||
}
|
||||
|
||||
// deleteObject container for Delete element in MultiObjects Delete XML request
|
||||
type deleteObject struct {
|
||||
Key string
|
||||
VersionID string `xml:"VersionId,omitempty"`
|
||||
}
|
||||
|
||||
// deletedObject container for Deleted element in MultiObjects Delete XML response
|
||||
type deletedObject struct {
|
||||
Key string
|
||||
VersionID string `xml:"VersionId,omitempty"`
|
||||
// These fields are ignored.
|
||||
DeleteMarker bool
|
||||
DeleteMarkerVersionID string `xml:"DeleteMarkerVersionId,omitempty"`
|
||||
}
|
||||
|
||||
// nonDeletedObject container for Error element (failed deletion) in MultiObjects Delete XML response
|
||||
type nonDeletedObject struct {
|
||||
Key string
|
||||
Code string
|
||||
Message string
|
||||
VersionID string `xml:"VersionId"`
|
||||
}
|
||||
|
||||
// deletedMultiObjects container for MultiObjects Delete XML request
|
||||
type deleteMultiObjects struct {
|
||||
XMLName xml.Name `xml:"Delete"`
|
||||
Quiet bool
|
||||
Objects []deleteObject `xml:"Object"`
|
||||
}
|
||||
|
||||
// deletedMultiObjectsResult container for MultiObjects Delete XML response
|
||||
type deleteMultiObjectsResult struct {
|
||||
XMLName xml.Name `xml:"DeleteResult"`
|
||||
DeletedObjects []deletedObject `xml:"Deleted"`
|
||||
UnDeletedObjects []nonDeletedObject `xml:"Error"`
|
||||
}
|
||||
+757
@@ -0,0 +1,757 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* (C) 2018-2020 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/binary"
|
||||
"encoding/xml"
|
||||
"errors"
|
||||
"fmt"
|
||||
"hash"
|
||||
"hash/crc32"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
|
||||
"github.com/minio/minio-go/v7/pkg/encrypt"
|
||||
"github.com/minio/minio-go/v7/pkg/s3utils"
|
||||
)
|
||||
|
||||
// CSVFileHeaderInfo - is the parameter for whether to utilize headers.
|
||||
type CSVFileHeaderInfo string
|
||||
|
||||
// Constants for file header info.
|
||||
const (
|
||||
CSVFileHeaderInfoNone CSVFileHeaderInfo = "NONE"
|
||||
CSVFileHeaderInfoIgnore CSVFileHeaderInfo = "IGNORE"
|
||||
CSVFileHeaderInfoUse CSVFileHeaderInfo = "USE"
|
||||
)
|
||||
|
||||
// SelectCompressionType - is the parameter for what type of compression is
|
||||
// present
|
||||
type SelectCompressionType string
|
||||
|
||||
// Constants for compression types under select API.
|
||||
const (
|
||||
SelectCompressionNONE SelectCompressionType = "NONE"
|
||||
SelectCompressionGZIP SelectCompressionType = "GZIP"
|
||||
SelectCompressionBZIP SelectCompressionType = "BZIP2"
|
||||
|
||||
// Non-standard compression schemes, supported by MinIO hosts:
|
||||
|
||||
SelectCompressionZSTD SelectCompressionType = "ZSTD" // Zstandard compression.
|
||||
SelectCompressionLZ4 SelectCompressionType = "LZ4" // LZ4 Stream
|
||||
SelectCompressionS2 SelectCompressionType = "S2" // S2 Stream
|
||||
SelectCompressionSNAPPY SelectCompressionType = "SNAPPY" // Snappy stream
|
||||
)
|
||||
|
||||
// CSVQuoteFields - is the parameter for how CSV fields are quoted.
|
||||
type CSVQuoteFields string
|
||||
|
||||
// Constants for csv quote styles.
|
||||
const (
|
||||
CSVQuoteFieldsAlways CSVQuoteFields = "Always"
|
||||
CSVQuoteFieldsAsNeeded CSVQuoteFields = "AsNeeded"
|
||||
)
|
||||
|
||||
// QueryExpressionType - is of what syntax the expression is, this should only
|
||||
// be SQL
|
||||
type QueryExpressionType string
|
||||
|
||||
// Constants for expression type.
|
||||
const (
|
||||
QueryExpressionTypeSQL QueryExpressionType = "SQL"
|
||||
)
|
||||
|
||||
// JSONType determines json input serialization type.
|
||||
type JSONType string
|
||||
|
||||
// Constants for JSONTypes.
|
||||
const (
|
||||
JSONDocumentType JSONType = "DOCUMENT"
|
||||
JSONLinesType JSONType = "LINES"
|
||||
)
|
||||
|
||||
// ParquetInputOptions parquet input specific options
|
||||
type ParquetInputOptions struct{}
|
||||
|
||||
// CSVInputOptions csv input specific options
|
||||
type CSVInputOptions struct {
|
||||
FileHeaderInfo CSVFileHeaderInfo
|
||||
fileHeaderInfoSet bool
|
||||
|
||||
RecordDelimiter string
|
||||
recordDelimiterSet bool
|
||||
|
||||
FieldDelimiter string
|
||||
fieldDelimiterSet bool
|
||||
|
||||
QuoteCharacter string
|
||||
quoteCharacterSet bool
|
||||
|
||||
QuoteEscapeCharacter string
|
||||
quoteEscapeCharacterSet bool
|
||||
|
||||
Comments string
|
||||
commentsSet bool
|
||||
}
|
||||
|
||||
// SetFileHeaderInfo sets the file header info in the CSV input options
|
||||
func (c *CSVInputOptions) SetFileHeaderInfo(val CSVFileHeaderInfo) {
|
||||
c.FileHeaderInfo = val
|
||||
c.fileHeaderInfoSet = true
|
||||
}
|
||||
|
||||
// SetRecordDelimiter sets the record delimiter in the CSV input options
|
||||
func (c *CSVInputOptions) SetRecordDelimiter(val string) {
|
||||
c.RecordDelimiter = val
|
||||
c.recordDelimiterSet = true
|
||||
}
|
||||
|
||||
// SetFieldDelimiter sets the field delimiter in the CSV input options
|
||||
func (c *CSVInputOptions) SetFieldDelimiter(val string) {
|
||||
c.FieldDelimiter = val
|
||||
c.fieldDelimiterSet = true
|
||||
}
|
||||
|
||||
// SetQuoteCharacter sets the quote character in the CSV input options
|
||||
func (c *CSVInputOptions) SetQuoteCharacter(val string) {
|
||||
c.QuoteCharacter = val
|
||||
c.quoteCharacterSet = true
|
||||
}
|
||||
|
||||
// SetQuoteEscapeCharacter sets the quote escape character in the CSV input options
|
||||
func (c *CSVInputOptions) SetQuoteEscapeCharacter(val string) {
|
||||
c.QuoteEscapeCharacter = val
|
||||
c.quoteEscapeCharacterSet = true
|
||||
}
|
||||
|
||||
// SetComments sets the comments character in the CSV input options
|
||||
func (c *CSVInputOptions) SetComments(val string) {
|
||||
c.Comments = val
|
||||
c.commentsSet = true
|
||||
}
|
||||
|
||||
// MarshalXML - produces the xml representation of the CSV input options struct
|
||||
func (c CSVInputOptions) MarshalXML(e *xml.Encoder, start xml.StartElement) error {
|
||||
if err := e.EncodeToken(start); err != nil {
|
||||
return err
|
||||
}
|
||||
if c.FileHeaderInfo != "" || c.fileHeaderInfoSet {
|
||||
if err := e.EncodeElement(c.FileHeaderInfo, xml.StartElement{Name: xml.Name{Local: "FileHeaderInfo"}}); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
if c.RecordDelimiter != "" || c.recordDelimiterSet {
|
||||
if err := e.EncodeElement(c.RecordDelimiter, xml.StartElement{Name: xml.Name{Local: "RecordDelimiter"}}); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
if c.FieldDelimiter != "" || c.fieldDelimiterSet {
|
||||
if err := e.EncodeElement(c.FieldDelimiter, xml.StartElement{Name: xml.Name{Local: "FieldDelimiter"}}); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
if c.QuoteCharacter != "" || c.quoteCharacterSet {
|
||||
if err := e.EncodeElement(c.QuoteCharacter, xml.StartElement{Name: xml.Name{Local: "QuoteCharacter"}}); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
if c.QuoteEscapeCharacter != "" || c.quoteEscapeCharacterSet {
|
||||
if err := e.EncodeElement(c.QuoteEscapeCharacter, xml.StartElement{Name: xml.Name{Local: "QuoteEscapeCharacter"}}); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
if c.Comments != "" || c.commentsSet {
|
||||
if err := e.EncodeElement(c.Comments, xml.StartElement{Name: xml.Name{Local: "Comments"}}); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return e.EncodeToken(xml.EndElement{Name: start.Name})
|
||||
}
|
||||
|
||||
// CSVOutputOptions csv output specific options
|
||||
type CSVOutputOptions struct {
|
||||
QuoteFields CSVQuoteFields
|
||||
quoteFieldsSet bool
|
||||
|
||||
RecordDelimiter string
|
||||
recordDelimiterSet bool
|
||||
|
||||
FieldDelimiter string
|
||||
fieldDelimiterSet bool
|
||||
|
||||
QuoteCharacter string
|
||||
quoteCharacterSet bool
|
||||
|
||||
QuoteEscapeCharacter string
|
||||
quoteEscapeCharacterSet bool
|
||||
}
|
||||
|
||||
// SetQuoteFields sets the quote field parameter in the CSV output options
|
||||
func (c *CSVOutputOptions) SetQuoteFields(val CSVQuoteFields) {
|
||||
c.QuoteFields = val
|
||||
c.quoteFieldsSet = true
|
||||
}
|
||||
|
||||
// SetRecordDelimiter sets the record delimiter character in the CSV output options
|
||||
func (c *CSVOutputOptions) SetRecordDelimiter(val string) {
|
||||
c.RecordDelimiter = val
|
||||
c.recordDelimiterSet = true
|
||||
}
|
||||
|
||||
// SetFieldDelimiter sets the field delimiter character in the CSV output options
|
||||
func (c *CSVOutputOptions) SetFieldDelimiter(val string) {
|
||||
c.FieldDelimiter = val
|
||||
c.fieldDelimiterSet = true
|
||||
}
|
||||
|
||||
// SetQuoteCharacter sets the quote character in the CSV output options
|
||||
func (c *CSVOutputOptions) SetQuoteCharacter(val string) {
|
||||
c.QuoteCharacter = val
|
||||
c.quoteCharacterSet = true
|
||||
}
|
||||
|
||||
// SetQuoteEscapeCharacter sets the quote escape character in the CSV output options
|
||||
func (c *CSVOutputOptions) SetQuoteEscapeCharacter(val string) {
|
||||
c.QuoteEscapeCharacter = val
|
||||
c.quoteEscapeCharacterSet = true
|
||||
}
|
||||
|
||||
// MarshalXML - produces the xml representation of the CSVOutputOptions struct
|
||||
func (c CSVOutputOptions) MarshalXML(e *xml.Encoder, start xml.StartElement) error {
|
||||
if err := e.EncodeToken(start); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if c.QuoteFields != "" || c.quoteFieldsSet {
|
||||
if err := e.EncodeElement(c.QuoteFields, xml.StartElement{Name: xml.Name{Local: "QuoteFields"}}); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
if c.RecordDelimiter != "" || c.recordDelimiterSet {
|
||||
if err := e.EncodeElement(c.RecordDelimiter, xml.StartElement{Name: xml.Name{Local: "RecordDelimiter"}}); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
if c.FieldDelimiter != "" || c.fieldDelimiterSet {
|
||||
if err := e.EncodeElement(c.FieldDelimiter, xml.StartElement{Name: xml.Name{Local: "FieldDelimiter"}}); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
if c.QuoteCharacter != "" || c.quoteCharacterSet {
|
||||
if err := e.EncodeElement(c.QuoteCharacter, xml.StartElement{Name: xml.Name{Local: "QuoteCharacter"}}); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
if c.QuoteEscapeCharacter != "" || c.quoteEscapeCharacterSet {
|
||||
if err := e.EncodeElement(c.QuoteEscapeCharacter, xml.StartElement{Name: xml.Name{Local: "QuoteEscapeCharacter"}}); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return e.EncodeToken(xml.EndElement{Name: start.Name})
|
||||
}
|
||||
|
||||
// JSONInputOptions json input specific options
|
||||
type JSONInputOptions struct {
|
||||
Type JSONType
|
||||
typeSet bool
|
||||
}
|
||||
|
||||
// SetType sets the JSON type in the JSON input options
|
||||
func (j *JSONInputOptions) SetType(typ JSONType) {
|
||||
j.Type = typ
|
||||
j.typeSet = true
|
||||
}
|
||||
|
||||
// MarshalXML - produces the xml representation of the JSONInputOptions struct
|
||||
func (j JSONInputOptions) MarshalXML(e *xml.Encoder, start xml.StartElement) error {
|
||||
if err := e.EncodeToken(start); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if j.Type != "" || j.typeSet {
|
||||
if err := e.EncodeElement(j.Type, xml.StartElement{Name: xml.Name{Local: "Type"}}); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return e.EncodeToken(xml.EndElement{Name: start.Name})
|
||||
}
|
||||
|
||||
// JSONOutputOptions - json output specific options
|
||||
type JSONOutputOptions struct {
|
||||
RecordDelimiter string
|
||||
recordDelimiterSet bool
|
||||
}
|
||||
|
||||
// SetRecordDelimiter sets the record delimiter in the JSON output options
|
||||
func (j *JSONOutputOptions) SetRecordDelimiter(val string) {
|
||||
j.RecordDelimiter = val
|
||||
j.recordDelimiterSet = true
|
||||
}
|
||||
|
||||
// MarshalXML - produces the xml representation of the JSONOutputOptions struct
|
||||
func (j JSONOutputOptions) MarshalXML(e *xml.Encoder, start xml.StartElement) error {
|
||||
if err := e.EncodeToken(start); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if j.RecordDelimiter != "" || j.recordDelimiterSet {
|
||||
if err := e.EncodeElement(j.RecordDelimiter, xml.StartElement{Name: xml.Name{Local: "RecordDelimiter"}}); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return e.EncodeToken(xml.EndElement{Name: start.Name})
|
||||
}
|
||||
|
||||
// SelectObjectInputSerialization - input serialization parameters
|
||||
type SelectObjectInputSerialization struct {
|
||||
CompressionType SelectCompressionType `xml:"CompressionType,omitempty"`
|
||||
Parquet *ParquetInputOptions `xml:"Parquet,omitempty"`
|
||||
CSV *CSVInputOptions `xml:"CSV,omitempty"`
|
||||
JSON *JSONInputOptions `xml:"JSON,omitempty"`
|
||||
}
|
||||
|
||||
// SelectObjectOutputSerialization - output serialization parameters.
|
||||
type SelectObjectOutputSerialization struct {
|
||||
CSV *CSVOutputOptions `xml:"CSV,omitempty"`
|
||||
JSON *JSONOutputOptions `xml:"JSON,omitempty"`
|
||||
}
|
||||
|
||||
// SelectObjectOptions - represents the input select body
|
||||
type SelectObjectOptions struct {
|
||||
XMLName xml.Name `xml:"SelectObjectContentRequest" json:"-"`
|
||||
ServerSideEncryption encrypt.ServerSide `xml:"-"`
|
||||
Expression string
|
||||
ExpressionType QueryExpressionType
|
||||
InputSerialization SelectObjectInputSerialization
|
||||
OutputSerialization SelectObjectOutputSerialization
|
||||
RequestProgress struct {
|
||||
Enabled bool
|
||||
}
|
||||
}
|
||||
|
||||
// Header returns the http.Header representation of the SelectObject options.
|
||||
func (o SelectObjectOptions) Header() http.Header {
|
||||
headers := make(http.Header)
|
||||
if o.ServerSideEncryption != nil && o.ServerSideEncryption.Type() == encrypt.SSEC {
|
||||
o.ServerSideEncryption.Marshal(headers)
|
||||
}
|
||||
return headers
|
||||
}
|
||||
|
||||
// SelectObjectType - is the parameter which defines what type of object the
|
||||
// operation is being performed on.
|
||||
type SelectObjectType string
|
||||
|
||||
// Constants for input data types.
|
||||
const (
|
||||
SelectObjectTypeCSV SelectObjectType = "CSV"
|
||||
SelectObjectTypeJSON SelectObjectType = "JSON"
|
||||
SelectObjectTypeParquet SelectObjectType = "Parquet"
|
||||
)
|
||||
|
||||
// preludeInfo is used for keeping track of necessary information from the
|
||||
// prelude.
|
||||
type preludeInfo struct {
|
||||
totalLen uint32
|
||||
headerLen uint32
|
||||
}
|
||||
|
||||
// SelectResults is used for the streaming responses from the server.
|
||||
type SelectResults struct {
|
||||
pipeReader *io.PipeReader
|
||||
resp *http.Response
|
||||
stats *StatsMessage
|
||||
progress *ProgressMessage
|
||||
}
|
||||
|
||||
// ProgressMessage is a struct for progress xml message.
|
||||
type ProgressMessage struct {
|
||||
XMLName xml.Name `xml:"Progress" json:"-"`
|
||||
StatsMessage
|
||||
}
|
||||
|
||||
// StatsMessage is a struct for stat xml message.
|
||||
type StatsMessage struct {
|
||||
XMLName xml.Name `xml:"Stats" json:"-"`
|
||||
BytesScanned int64
|
||||
BytesProcessed int64
|
||||
BytesReturned int64
|
||||
}
|
||||
|
||||
// messageType represents the type of message.
|
||||
type messageType string
|
||||
|
||||
const (
|
||||
errorMsg messageType = "error"
|
||||
commonMsg messageType = "event"
|
||||
)
|
||||
|
||||
// eventType represents the type of event.
|
||||
type eventType string
|
||||
|
||||
// list of event-types returned by Select API.
|
||||
const (
|
||||
endEvent eventType = "End"
|
||||
recordsEvent eventType = "Records"
|
||||
progressEvent eventType = "Progress"
|
||||
statsEvent eventType = "Stats"
|
||||
)
|
||||
|
||||
// contentType represents content type of event.
|
||||
type contentType string
|
||||
|
||||
const (
|
||||
xmlContent contentType = "text/xml"
|
||||
)
|
||||
|
||||
// SelectObjectContent is a implementation of http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectSELECTContent.html AWS S3 API.
|
||||
func (c *Client) SelectObjectContent(ctx context.Context, bucketName, objectName string, opts SelectObjectOptions) (*SelectResults, error) {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := s3utils.CheckValidObjectName(objectName); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
selectReqBytes, err := xml.Marshal(opts)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("select", "")
|
||||
urlValues.Set("select-type", "2")
|
||||
|
||||
// Execute POST on bucket/object.
|
||||
resp, err := c.executeMethod(ctx, http.MethodPost, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
objectName: objectName,
|
||||
queryValues: urlValues,
|
||||
customHeader: opts.Header(),
|
||||
contentMD5Base64: sumMD5Base64(selectReqBytes),
|
||||
contentSHA256Hex: sum256Hex(selectReqBytes),
|
||||
contentBody: bytes.NewReader(selectReqBytes),
|
||||
contentLength: int64(len(selectReqBytes)),
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return NewSelectResults(resp, bucketName)
|
||||
}
|
||||
|
||||
// NewSelectResults creates a Select Result parser that parses the response
|
||||
// and returns a Reader that will return parsed and assembled select output.
|
||||
func NewSelectResults(resp *http.Response, bucketName string) (*SelectResults, error) {
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return nil, httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
|
||||
pipeReader, pipeWriter := io.Pipe()
|
||||
streamer := &SelectResults{
|
||||
resp: resp,
|
||||
stats: &StatsMessage{},
|
||||
progress: &ProgressMessage{},
|
||||
pipeReader: pipeReader,
|
||||
}
|
||||
streamer.start(pipeWriter)
|
||||
return streamer, nil
|
||||
}
|
||||
|
||||
// Close - closes the underlying response body and the stream reader.
|
||||
func (s *SelectResults) Close() error {
|
||||
defer closeResponse(s.resp)
|
||||
return s.pipeReader.Close()
|
||||
}
|
||||
|
||||
// Read - is a reader compatible implementation for SelectObjectContent records.
|
||||
func (s *SelectResults) Read(b []byte) (n int, err error) {
|
||||
return s.pipeReader.Read(b)
|
||||
}
|
||||
|
||||
// Stats - information about a request's stats when processing is complete.
|
||||
func (s *SelectResults) Stats() *StatsMessage {
|
||||
return s.stats
|
||||
}
|
||||
|
||||
// Progress - information about the progress of a request.
|
||||
func (s *SelectResults) Progress() *ProgressMessage {
|
||||
return s.progress
|
||||
}
|
||||
|
||||
// start is the main function that decodes the large byte array into
|
||||
// several events that are sent through the eventstream.
|
||||
func (s *SelectResults) start(pipeWriter *io.PipeWriter) {
|
||||
go func() {
|
||||
for {
|
||||
var prelude preludeInfo
|
||||
headers := make(http.Header)
|
||||
var err error
|
||||
|
||||
// Create CRC code
|
||||
crc := crc32.New(crc32.IEEETable)
|
||||
crcReader := io.TeeReader(s.resp.Body, crc)
|
||||
|
||||
// Extract the prelude(12 bytes) into a struct to extract relevant information.
|
||||
prelude, err = processPrelude(crcReader, crc)
|
||||
if err != nil {
|
||||
pipeWriter.CloseWithError(err)
|
||||
closeResponse(s.resp)
|
||||
return
|
||||
}
|
||||
|
||||
// Extract the headers(variable bytes) into a struct to extract relevant information
|
||||
if prelude.headerLen > 0 {
|
||||
if err = extractHeader(io.LimitReader(crcReader, int64(prelude.headerLen)), headers); err != nil {
|
||||
pipeWriter.CloseWithError(err)
|
||||
closeResponse(s.resp)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// Get the actual payload length so that the appropriate amount of
|
||||
// bytes can be read or parsed.
|
||||
payloadLen := prelude.PayloadLen()
|
||||
|
||||
m := messageType(headers.Get("message-type"))
|
||||
|
||||
switch m {
|
||||
case errorMsg:
|
||||
pipeWriter.CloseWithError(errors.New(headers.Get("error-code") + ":\"" + headers.Get("error-message") + "\""))
|
||||
closeResponse(s.resp)
|
||||
return
|
||||
case commonMsg:
|
||||
// Get content-type of the payload.
|
||||
c := contentType(headers.Get("content-type"))
|
||||
|
||||
// Get event type of the payload.
|
||||
e := eventType(headers.Get("event-type"))
|
||||
|
||||
// Handle all supported events.
|
||||
switch e {
|
||||
case endEvent:
|
||||
pipeWriter.Close()
|
||||
closeResponse(s.resp)
|
||||
return
|
||||
case recordsEvent:
|
||||
if _, err = io.Copy(pipeWriter, io.LimitReader(crcReader, payloadLen)); err != nil {
|
||||
pipeWriter.CloseWithError(err)
|
||||
closeResponse(s.resp)
|
||||
return
|
||||
}
|
||||
case progressEvent:
|
||||
switch c {
|
||||
case xmlContent:
|
||||
if err = xmlDecoder(io.LimitReader(crcReader, payloadLen), s.progress); err != nil {
|
||||
pipeWriter.CloseWithError(err)
|
||||
closeResponse(s.resp)
|
||||
return
|
||||
}
|
||||
default:
|
||||
pipeWriter.CloseWithError(fmt.Errorf("Unexpected content-type %s sent for event-type %s", c, progressEvent))
|
||||
closeResponse(s.resp)
|
||||
return
|
||||
}
|
||||
case statsEvent:
|
||||
switch c {
|
||||
case xmlContent:
|
||||
if err = xmlDecoder(io.LimitReader(crcReader, payloadLen), s.stats); err != nil {
|
||||
pipeWriter.CloseWithError(err)
|
||||
closeResponse(s.resp)
|
||||
return
|
||||
}
|
||||
default:
|
||||
pipeWriter.CloseWithError(fmt.Errorf("Unexpected content-type %s sent for event-type %s", c, statsEvent))
|
||||
closeResponse(s.resp)
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Ensures that the full message's CRC is correct and
|
||||
// that the message is not corrupted
|
||||
if err := checkCRC(s.resp.Body, crc.Sum32()); err != nil {
|
||||
pipeWriter.CloseWithError(err)
|
||||
closeResponse(s.resp)
|
||||
return
|
||||
}
|
||||
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
// PayloadLen is a function that calculates the length of the payload.
|
||||
func (p preludeInfo) PayloadLen() int64 {
|
||||
return int64(p.totalLen - p.headerLen - 16)
|
||||
}
|
||||
|
||||
// processPrelude is the function that reads the 12 bytes of the prelude and
|
||||
// ensures the CRC is correct while also extracting relevant information into
|
||||
// the struct,
|
||||
func processPrelude(prelude io.Reader, crc hash.Hash32) (preludeInfo, error) {
|
||||
var err error
|
||||
pInfo := preludeInfo{}
|
||||
|
||||
// reads total length of the message (first 4 bytes)
|
||||
pInfo.totalLen, err = extractUint32(prelude)
|
||||
if err != nil {
|
||||
return pInfo, err
|
||||
}
|
||||
|
||||
// reads total header length of the message (2nd 4 bytes)
|
||||
pInfo.headerLen, err = extractUint32(prelude)
|
||||
if err != nil {
|
||||
return pInfo, err
|
||||
}
|
||||
|
||||
// checks that the CRC is correct (3rd 4 bytes)
|
||||
preCRC := crc.Sum32()
|
||||
if err := checkCRC(prelude, preCRC); err != nil {
|
||||
return pInfo, err
|
||||
}
|
||||
|
||||
return pInfo, nil
|
||||
}
|
||||
|
||||
// extracts the relevant information from the Headers.
|
||||
func extractHeader(body io.Reader, myHeaders http.Header) error {
|
||||
for {
|
||||
// extracts the first part of the header,
|
||||
headerTypeName, err := extractHeaderType(body)
|
||||
if err != nil {
|
||||
// Since end of file, we have read all of our headers
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
// reads the 7 present in the header and ignores it.
|
||||
extractUint8(body)
|
||||
|
||||
headerValueName, err := extractHeaderValue(body)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
myHeaders.Set(headerTypeName, headerValueName)
|
||||
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// extractHeaderType extracts the first half of the header message, the header type.
|
||||
func extractHeaderType(body io.Reader) (string, error) {
|
||||
// extracts 2 bit integer
|
||||
headerNameLen, err := extractUint8(body)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
// extracts the string with the appropriate number of bytes
|
||||
headerName, err := extractString(body, int(headerNameLen))
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return strings.TrimPrefix(headerName, ":"), nil
|
||||
}
|
||||
|
||||
// extractsHeaderValue extracts the second half of the header message, the
|
||||
// header value
|
||||
func extractHeaderValue(body io.Reader) (string, error) {
|
||||
bodyLen, err := extractUint16(body)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
bodyName, err := extractString(body, int(bodyLen))
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return bodyName, nil
|
||||
}
|
||||
|
||||
// extracts a string from byte array of a particular number of bytes.
|
||||
func extractString(source io.Reader, lenBytes int) (string, error) {
|
||||
myVal := make([]byte, lenBytes)
|
||||
_, err := source.Read(myVal)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return string(myVal), nil
|
||||
}
|
||||
|
||||
// extractUint32 extracts a 4 byte integer from the byte array.
|
||||
func extractUint32(r io.Reader) (uint32, error) {
|
||||
buf := make([]byte, 4)
|
||||
_, err := readFull(r, buf)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return binary.BigEndian.Uint32(buf), nil
|
||||
}
|
||||
|
||||
// extractUint16 extracts a 2 byte integer from the byte array.
|
||||
func extractUint16(r io.Reader) (uint16, error) {
|
||||
buf := make([]byte, 2)
|
||||
_, err := readFull(r, buf)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return binary.BigEndian.Uint16(buf), nil
|
||||
}
|
||||
|
||||
// extractUint8 extracts a 1 byte integer from the byte array.
|
||||
func extractUint8(r io.Reader) (uint8, error) {
|
||||
buf := make([]byte, 1)
|
||||
_, err := readFull(r, buf)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return buf[0], nil
|
||||
}
|
||||
|
||||
// checkCRC ensures that the CRC matches with the one from the reader.
|
||||
func checkCRC(r io.Reader, expect uint32) error {
|
||||
msgCRC, err := extractUint32(r)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if msgCRC != expect {
|
||||
return fmt.Errorf("Checksum Mismatch, MessageCRC of 0x%X does not equal expected CRC of 0x%X", msgCRC, expect)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
+124
@@ -0,0 +1,124 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2015-2020 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
|
||||
"github.com/minio/minio-go/v7/pkg/s3utils"
|
||||
)
|
||||
|
||||
// BucketExists verifies if bucket exists and you have permission to access it. Allows for a Context to
|
||||
// control cancellations and timeouts.
|
||||
func (c *Client) BucketExists(ctx context.Context, bucketName string) (bool, error) {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return false, err
|
||||
}
|
||||
|
||||
// Execute HEAD on bucketName.
|
||||
resp, err := c.executeMethod(ctx, http.MethodHead, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
contentSHA256Hex: emptySHA256Hex,
|
||||
})
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
if ToErrorResponse(err).Code == "NoSuchBucket" {
|
||||
return false, nil
|
||||
}
|
||||
return false, err
|
||||
}
|
||||
if resp != nil {
|
||||
resperr := httpRespToErrorResponse(resp, bucketName, "")
|
||||
if ToErrorResponse(resperr).Code == "NoSuchBucket" {
|
||||
return false, nil
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return false, httpRespToErrorResponse(resp, bucketName, "")
|
||||
}
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// StatObject verifies if object exists, you have permission to access it
|
||||
// and returns information about the object.
|
||||
func (c *Client) StatObject(ctx context.Context, bucketName, objectName string, opts StatObjectOptions) (ObjectInfo, error) {
|
||||
// Input validation.
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return ObjectInfo{}, ErrorResponse{
|
||||
StatusCode: http.StatusBadRequest,
|
||||
Code: "InvalidBucketName",
|
||||
Message: err.Error(),
|
||||
}
|
||||
}
|
||||
if err := s3utils.CheckValidObjectName(objectName); err != nil {
|
||||
return ObjectInfo{}, ErrorResponse{
|
||||
StatusCode: http.StatusBadRequest,
|
||||
Code: "XMinioInvalidObjectName",
|
||||
Message: err.Error(),
|
||||
}
|
||||
}
|
||||
headers := opts.Header()
|
||||
if opts.Internal.ReplicationDeleteMarker {
|
||||
headers.Set(minIOBucketReplicationDeleteMarker, "true")
|
||||
}
|
||||
if opts.Internal.IsReplicationReadyForDeleteMarker {
|
||||
headers.Set(isMinioTgtReplicationReady, "true")
|
||||
}
|
||||
|
||||
// Execute HEAD on objectName.
|
||||
resp, err := c.executeMethod(ctx, http.MethodHead, requestMetadata{
|
||||
bucketName: bucketName,
|
||||
objectName: objectName,
|
||||
queryValues: opts.toQueryValues(),
|
||||
contentSHA256Hex: emptySHA256Hex,
|
||||
customHeader: headers,
|
||||
})
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return ObjectInfo{}, err
|
||||
}
|
||||
|
||||
if resp != nil {
|
||||
deleteMarker := resp.Header.Get(amzDeleteMarker) == "true"
|
||||
replicationReady := resp.Header.Get(minioTgtReplicationReady) == "true"
|
||||
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusPartialContent {
|
||||
if resp.StatusCode == http.StatusMethodNotAllowed && opts.VersionID != "" && deleteMarker {
|
||||
errResp := ErrorResponse{
|
||||
StatusCode: resp.StatusCode,
|
||||
Code: "MethodNotAllowed",
|
||||
Message: "The specified method is not allowed against this resource.",
|
||||
BucketName: bucketName,
|
||||
Key: objectName,
|
||||
}
|
||||
return ObjectInfo{
|
||||
VersionID: resp.Header.Get(amzVersionID),
|
||||
IsDeleteMarker: deleteMarker,
|
||||
}, errResp
|
||||
}
|
||||
return ObjectInfo{
|
||||
VersionID: resp.Header.Get(amzVersionID),
|
||||
IsDeleteMarker: deleteMarker,
|
||||
ReplicationReady: replicationReady, // whether delete marker can be replicated
|
||||
}, httpRespToErrorResponse(resp, bucketName, objectName)
|
||||
}
|
||||
}
|
||||
|
||||
return ToObjectInfo(bucketName, objectName, resp.Header)
|
||||
}
|
||||
+1032
File diff suppressed because it is too large
Load Diff
+256
@@ -0,0 +1,256 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2015-2017 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"path"
|
||||
"sync"
|
||||
|
||||
"github.com/minio/minio-go/v7/pkg/credentials"
|
||||
"github.com/minio/minio-go/v7/pkg/s3utils"
|
||||
"github.com/minio/minio-go/v7/pkg/signer"
|
||||
)
|
||||
|
||||
// bucketLocationCache - Provides simple mechanism to hold bucket
|
||||
// locations in memory.
|
||||
type bucketLocationCache struct {
|
||||
// mutex is used for handling the concurrent
|
||||
// read/write requests for cache.
|
||||
sync.RWMutex
|
||||
|
||||
// items holds the cached bucket locations.
|
||||
items map[string]string
|
||||
}
|
||||
|
||||
// newBucketLocationCache - Provides a new bucket location cache to be
|
||||
// used internally with the client object.
|
||||
func newBucketLocationCache() *bucketLocationCache {
|
||||
return &bucketLocationCache{
|
||||
items: make(map[string]string),
|
||||
}
|
||||
}
|
||||
|
||||
// Get - Returns a value of a given key if it exists.
|
||||
func (r *bucketLocationCache) Get(bucketName string) (location string, ok bool) {
|
||||
r.RLock()
|
||||
defer r.RUnlock()
|
||||
location, ok = r.items[bucketName]
|
||||
return
|
||||
}
|
||||
|
||||
// Set - Will persist a value into cache.
|
||||
func (r *bucketLocationCache) Set(bucketName, location string) {
|
||||
r.Lock()
|
||||
defer r.Unlock()
|
||||
r.items[bucketName] = location
|
||||
}
|
||||
|
||||
// Delete - Deletes a bucket name from cache.
|
||||
func (r *bucketLocationCache) Delete(bucketName string) {
|
||||
r.Lock()
|
||||
defer r.Unlock()
|
||||
delete(r.items, bucketName)
|
||||
}
|
||||
|
||||
// GetBucketLocation - get location for the bucket name from location cache, if not
|
||||
// fetch freshly by making a new request.
|
||||
func (c *Client) GetBucketLocation(ctx context.Context, bucketName string) (string, error) {
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return c.getBucketLocation(ctx, bucketName)
|
||||
}
|
||||
|
||||
// getBucketLocation - Get location for the bucketName from location map cache, if not
|
||||
// fetch freshly by making a new request.
|
||||
func (c *Client) getBucketLocation(ctx context.Context, bucketName string) (string, error) {
|
||||
if err := s3utils.CheckValidBucketName(bucketName); err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
// Region set then no need to fetch bucket location.
|
||||
if c.region != "" {
|
||||
return c.region, nil
|
||||
}
|
||||
|
||||
if location, ok := c.bucketLocCache.Get(bucketName); ok {
|
||||
return location, nil
|
||||
}
|
||||
|
||||
// Initialize a new request.
|
||||
req, err := c.getBucketLocationRequest(ctx, bucketName)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
// Initiate the request.
|
||||
resp, err := c.do(req)
|
||||
defer closeResponse(resp)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
location, err := processBucketLocationResponse(resp, bucketName)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
c.bucketLocCache.Set(bucketName, location)
|
||||
return location, nil
|
||||
}
|
||||
|
||||
// processes the getBucketLocation http response from the server.
|
||||
func processBucketLocationResponse(resp *http.Response, bucketName string) (bucketLocation string, err error) {
|
||||
if resp != nil {
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
err = httpRespToErrorResponse(resp, bucketName, "")
|
||||
errResp := ToErrorResponse(err)
|
||||
// For access denied error, it could be an anonymous
|
||||
// request. Move forward and let the top level callers
|
||||
// succeed if possible based on their policy.
|
||||
switch errResp.Code {
|
||||
case "NotImplemented":
|
||||
switch errResp.Server {
|
||||
case "AmazonSnowball":
|
||||
return "snowball", nil
|
||||
case "cloudflare":
|
||||
return "us-east-1", nil
|
||||
}
|
||||
case "AuthorizationHeaderMalformed":
|
||||
fallthrough
|
||||
case "InvalidRegion":
|
||||
fallthrough
|
||||
case "AccessDenied":
|
||||
if errResp.Region == "" {
|
||||
return "us-east-1", nil
|
||||
}
|
||||
return errResp.Region, nil
|
||||
}
|
||||
return "", err
|
||||
}
|
||||
}
|
||||
|
||||
// Extract location.
|
||||
var locationConstraint string
|
||||
err = xmlDecoder(resp.Body, &locationConstraint)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
location := locationConstraint
|
||||
// Location is empty will be 'us-east-1'.
|
||||
if location == "" {
|
||||
location = "us-east-1"
|
||||
}
|
||||
|
||||
// Location can be 'EU' convert it to meaningful 'eu-west-1'.
|
||||
if location == "EU" {
|
||||
location = "eu-west-1"
|
||||
}
|
||||
|
||||
// Save the location into cache.
|
||||
|
||||
// Return.
|
||||
return location, nil
|
||||
}
|
||||
|
||||
// getBucketLocationRequest - Wrapper creates a new getBucketLocation request.
|
||||
func (c *Client) getBucketLocationRequest(ctx context.Context, bucketName string) (*http.Request, error) {
|
||||
// Set location query.
|
||||
urlValues := make(url.Values)
|
||||
urlValues.Set("location", "")
|
||||
|
||||
// Set get bucket location always as path style.
|
||||
targetURL := *c.endpointURL
|
||||
|
||||
// as it works in makeTargetURL method from api.go file
|
||||
if h, p, err := net.SplitHostPort(targetURL.Host); err == nil {
|
||||
if targetURL.Scheme == "http" && p == "80" || targetURL.Scheme == "https" && p == "443" {
|
||||
targetURL.Host = h
|
||||
if ip := net.ParseIP(h); ip != nil && ip.To16() != nil {
|
||||
targetURL.Host = "[" + h + "]"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
isVirtualStyle := c.isVirtualHostStyleRequest(targetURL, bucketName)
|
||||
|
||||
var urlStr string
|
||||
|
||||
if isVirtualStyle {
|
||||
urlStr = c.endpointURL.Scheme + "://" + bucketName + "." + targetURL.Host + "/?location"
|
||||
} else {
|
||||
targetURL.Path = path.Join(bucketName, "") + "/"
|
||||
targetURL.RawQuery = urlValues.Encode()
|
||||
urlStr = targetURL.String()
|
||||
}
|
||||
|
||||
// Get a new HTTP request for the method.
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, urlStr, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Set UserAgent for the request.
|
||||
c.setUserAgent(req)
|
||||
|
||||
// Get credentials from the configured credentials provider.
|
||||
value, err := c.credsProvider.GetWithContext(c.CredContext())
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var (
|
||||
signerType = value.SignerType
|
||||
accessKeyID = value.AccessKeyID
|
||||
secretAccessKey = value.SecretAccessKey
|
||||
sessionToken = value.SessionToken
|
||||
)
|
||||
|
||||
// Custom signer set then override the behavior.
|
||||
if c.overrideSignerType != credentials.SignatureDefault {
|
||||
signerType = c.overrideSignerType
|
||||
}
|
||||
|
||||
// If signerType returned by credentials helper is anonymous,
|
||||
// then do not sign regardless of signerType override.
|
||||
if value.SignerType == credentials.SignatureAnonymous {
|
||||
signerType = credentials.SignatureAnonymous
|
||||
}
|
||||
|
||||
if signerType.IsAnonymous() {
|
||||
return req, nil
|
||||
}
|
||||
|
||||
if signerType.IsV2() {
|
||||
req = signer.SignV2(*req, accessKeyID, secretAccessKey, isVirtualStyle)
|
||||
return req, nil
|
||||
}
|
||||
|
||||
// Set sha256 sum for signature calculation only with signature version '4'.
|
||||
contentSha256 := emptySHA256Hex
|
||||
if c.secure {
|
||||
contentSha256 = unsignedPayload
|
||||
}
|
||||
|
||||
req.Header.Set("X-Amz-Content-Sha256", contentSha256)
|
||||
req = signer.SignV4(*req, accessKeyID, secretAccessKey, sessionToken, "us-east-1")
|
||||
return req, nil
|
||||
}
|
||||
+421
@@ -0,0 +1,421 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2015-2023 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"crypto/sha1"
|
||||
"crypto/sha256"
|
||||
"encoding/base64"
|
||||
"encoding/binary"
|
||||
"errors"
|
||||
"hash"
|
||||
"hash/crc32"
|
||||
"hash/crc64"
|
||||
"io"
|
||||
"math/bits"
|
||||
"net/http"
|
||||
"sort"
|
||||
)
|
||||
|
||||
// ChecksumType contains information about the checksum type.
|
||||
type ChecksumType uint32
|
||||
|
||||
const (
|
||||
|
||||
// ChecksumSHA256 indicates a SHA256 checksum.
|
||||
ChecksumSHA256 ChecksumType = 1 << iota
|
||||
// ChecksumSHA1 indicates a SHA-1 checksum.
|
||||
ChecksumSHA1
|
||||
// ChecksumCRC32 indicates a CRC32 checksum with IEEE table.
|
||||
ChecksumCRC32
|
||||
// ChecksumCRC32C indicates a CRC32 checksum with Castagnoli table.
|
||||
ChecksumCRC32C
|
||||
// ChecksumCRC64NVME indicates CRC64 with 0xad93d23594c93659 polynomial.
|
||||
ChecksumCRC64NVME
|
||||
|
||||
// Keep after all valid checksums
|
||||
checksumLast
|
||||
|
||||
// ChecksumFullObject is a modifier that can be used on CRC32 and CRC32C
|
||||
// to indicate full object checksums.
|
||||
ChecksumFullObject
|
||||
|
||||
// checksumMask is a mask for valid checksum types.
|
||||
checksumMask = checksumLast - 1
|
||||
|
||||
// ChecksumNone indicates no checksum.
|
||||
ChecksumNone ChecksumType = 0
|
||||
|
||||
// ChecksumFullObjectCRC32 indicates full object CRC32
|
||||
ChecksumFullObjectCRC32 = ChecksumCRC32 | ChecksumFullObject
|
||||
|
||||
// ChecksumFullObjectCRC32C indicates full object CRC32C
|
||||
ChecksumFullObjectCRC32C = ChecksumCRC32C | ChecksumFullObject
|
||||
|
||||
amzChecksumAlgo = "x-amz-checksum-algorithm"
|
||||
amzChecksumCRC32 = "x-amz-checksum-crc32"
|
||||
amzChecksumCRC32C = "x-amz-checksum-crc32c"
|
||||
amzChecksumSHA1 = "x-amz-checksum-sha1"
|
||||
amzChecksumSHA256 = "x-amz-checksum-sha256"
|
||||
amzChecksumCRC64NVME = "x-amz-checksum-crc64nvme"
|
||||
)
|
||||
|
||||
// Base returns the base type, without modifiers.
|
||||
func (c ChecksumType) Base() ChecksumType {
|
||||
return c & checksumMask
|
||||
}
|
||||
|
||||
// Is returns if c is all of t.
|
||||
func (c ChecksumType) Is(t ChecksumType) bool {
|
||||
return c&t == t
|
||||
}
|
||||
|
||||
// Key returns the header key.
|
||||
// returns empty string if invalid or none.
|
||||
func (c ChecksumType) Key() string {
|
||||
switch c & checksumMask {
|
||||
case ChecksumCRC32:
|
||||
return amzChecksumCRC32
|
||||
case ChecksumCRC32C:
|
||||
return amzChecksumCRC32C
|
||||
case ChecksumSHA1:
|
||||
return amzChecksumSHA1
|
||||
case ChecksumSHA256:
|
||||
return amzChecksumSHA256
|
||||
case ChecksumCRC64NVME:
|
||||
return amzChecksumCRC64NVME
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// CanComposite will return if the checksum type can be used for composite multipart upload on AWS.
|
||||
func (c ChecksumType) CanComposite() bool {
|
||||
switch c & checksumMask {
|
||||
case ChecksumSHA256, ChecksumSHA1, ChecksumCRC32, ChecksumCRC32C:
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// CanMergeCRC will return if the checksum type can be used for multipart upload on AWS.
|
||||
func (c ChecksumType) CanMergeCRC() bool {
|
||||
switch c & checksumMask {
|
||||
case ChecksumCRC32, ChecksumCRC32C, ChecksumCRC64NVME:
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// FullObjectRequested will return if the checksum type indicates full object checksum was requested.
|
||||
func (c ChecksumType) FullObjectRequested() bool {
|
||||
switch c & (ChecksumFullObject | checksumMask) {
|
||||
case ChecksumFullObjectCRC32C, ChecksumFullObjectCRC32, ChecksumCRC64NVME:
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// KeyCapitalized returns the capitalized key as used in HTTP headers.
|
||||
func (c ChecksumType) KeyCapitalized() string {
|
||||
return http.CanonicalHeaderKey(c.Key())
|
||||
}
|
||||
|
||||
// RawByteLen returns the size of the un-encoded checksum.
|
||||
func (c ChecksumType) RawByteLen() int {
|
||||
switch c & checksumMask {
|
||||
case ChecksumCRC32, ChecksumCRC32C:
|
||||
return 4
|
||||
case ChecksumSHA1:
|
||||
return sha1.Size
|
||||
case ChecksumSHA256:
|
||||
return sha256.Size
|
||||
case ChecksumCRC64NVME:
|
||||
return crc64.Size
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
const crc64NVMEPolynomial = 0xad93d23594c93659
|
||||
|
||||
// crc64 uses reversed polynomials.
|
||||
var crc64Table = crc64.MakeTable(bits.Reverse64(crc64NVMEPolynomial))
|
||||
|
||||
// Hasher returns a hasher corresponding to the checksum type.
|
||||
// Returns nil if no checksum.
|
||||
func (c ChecksumType) Hasher() hash.Hash {
|
||||
switch c & checksumMask {
|
||||
case ChecksumCRC32:
|
||||
return crc32.NewIEEE()
|
||||
case ChecksumCRC32C:
|
||||
return crc32.New(crc32.MakeTable(crc32.Castagnoli))
|
||||
case ChecksumSHA1:
|
||||
return sha1.New()
|
||||
case ChecksumSHA256:
|
||||
return sha256.New()
|
||||
case ChecksumCRC64NVME:
|
||||
return crc64.New(crc64Table)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// IsSet returns whether the type is valid and known.
|
||||
func (c ChecksumType) IsSet() bool {
|
||||
return bits.OnesCount32(uint32(c&checksumMask)) == 1
|
||||
}
|
||||
|
||||
// SetDefault will set the checksum if not already set.
|
||||
func (c *ChecksumType) SetDefault(t ChecksumType) {
|
||||
if !c.IsSet() {
|
||||
*c = t
|
||||
}
|
||||
}
|
||||
|
||||
// EncodeToString the encoded hash value of the content provided in b.
|
||||
func (c ChecksumType) EncodeToString(b []byte) string {
|
||||
if !c.IsSet() {
|
||||
return ""
|
||||
}
|
||||
h := c.Hasher()
|
||||
h.Write(b)
|
||||
return base64.StdEncoding.EncodeToString(h.Sum(nil))
|
||||
}
|
||||
|
||||
// String returns the type as a string.
|
||||
// CRC32, CRC32C, SHA1, and SHA256 for valid values.
|
||||
// Empty string for unset and "<invalid>" if not valid.
|
||||
func (c ChecksumType) String() string {
|
||||
switch c & checksumMask {
|
||||
case ChecksumCRC32:
|
||||
return "CRC32"
|
||||
case ChecksumCRC32C:
|
||||
return "CRC32C"
|
||||
case ChecksumSHA1:
|
||||
return "SHA1"
|
||||
case ChecksumSHA256:
|
||||
return "SHA256"
|
||||
case ChecksumNone:
|
||||
return ""
|
||||
case ChecksumCRC64NVME:
|
||||
return "CRC64NVME"
|
||||
}
|
||||
return "<invalid>"
|
||||
}
|
||||
|
||||
// ChecksumReader reads all of r and returns a checksum of type c.
|
||||
// Returns any error that may have occurred while reading.
|
||||
func (c ChecksumType) ChecksumReader(r io.Reader) (Checksum, error) {
|
||||
h := c.Hasher()
|
||||
if h == nil {
|
||||
return Checksum{}, nil
|
||||
}
|
||||
_, err := io.Copy(h, r)
|
||||
if err != nil {
|
||||
return Checksum{}, err
|
||||
}
|
||||
return NewChecksum(c, h.Sum(nil)), nil
|
||||
}
|
||||
|
||||
// ChecksumBytes returns a checksum of the content b with type c.
|
||||
func (c ChecksumType) ChecksumBytes(b []byte) Checksum {
|
||||
h := c.Hasher()
|
||||
if h == nil {
|
||||
return Checksum{}
|
||||
}
|
||||
n, err := h.Write(b)
|
||||
if err != nil || n != len(b) {
|
||||
// Shouldn't happen with these checksummers.
|
||||
return Checksum{}
|
||||
}
|
||||
return NewChecksum(c, h.Sum(nil))
|
||||
}
|
||||
|
||||
// Checksum is a type and encoded value.
|
||||
type Checksum struct {
|
||||
Type ChecksumType
|
||||
r []byte
|
||||
}
|
||||
|
||||
// NewChecksum sets the checksum to the value of b,
|
||||
// which is the raw hash output.
|
||||
// If the length of c does not match t.RawByteLen,
|
||||
// a checksum with ChecksumNone is returned.
|
||||
func NewChecksum(t ChecksumType, b []byte) Checksum {
|
||||
if t.IsSet() && len(b) == t.RawByteLen() {
|
||||
return Checksum{Type: t, r: b}
|
||||
}
|
||||
return Checksum{}
|
||||
}
|
||||
|
||||
// NewChecksumString sets the checksum to the value of s,
|
||||
// which is the base 64 encoded raw hash output.
|
||||
// If the length of c does not match t.RawByteLen, it is not added.
|
||||
func NewChecksumString(t ChecksumType, s string) Checksum {
|
||||
b, _ := base64.StdEncoding.DecodeString(s)
|
||||
if t.IsSet() && len(b) == t.RawByteLen() {
|
||||
return Checksum{Type: t, r: b}
|
||||
}
|
||||
return Checksum{}
|
||||
}
|
||||
|
||||
// IsSet returns whether the checksum is valid and known.
|
||||
func (c Checksum) IsSet() bool {
|
||||
return c.Type.IsSet() && len(c.r) == c.Type.RawByteLen()
|
||||
}
|
||||
|
||||
// Encoded returns the encoded value.
|
||||
// Returns the empty string if not set or valid.
|
||||
func (c Checksum) Encoded() string {
|
||||
if !c.IsSet() {
|
||||
return ""
|
||||
}
|
||||
return base64.StdEncoding.EncodeToString(c.r)
|
||||
}
|
||||
|
||||
// Raw returns the raw checksum value if set.
|
||||
func (c Checksum) Raw() []byte {
|
||||
if !c.IsSet() {
|
||||
return nil
|
||||
}
|
||||
return c.r
|
||||
}
|
||||
|
||||
// CompositeChecksum returns the composite checksum of all provided parts.
|
||||
func (c ChecksumType) CompositeChecksum(p []ObjectPart) (*Checksum, error) {
|
||||
if !c.CanComposite() {
|
||||
return nil, errors.New("cannot do composite checksum")
|
||||
}
|
||||
sort.Slice(p, func(i, j int) bool {
|
||||
return p[i].PartNumber < p[j].PartNumber
|
||||
})
|
||||
c = c.Base()
|
||||
crcBytes := make([]byte, 0, len(p)*c.RawByteLen())
|
||||
for _, part := range p {
|
||||
pCrc, err := part.ChecksumRaw(c)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
crcBytes = append(crcBytes, pCrc...)
|
||||
}
|
||||
h := c.Hasher()
|
||||
h.Write(crcBytes)
|
||||
return &Checksum{Type: c, r: h.Sum(nil)}, nil
|
||||
}
|
||||
|
||||
// FullObjectChecksum will return the full object checksum from provided parts.
|
||||
func (c ChecksumType) FullObjectChecksum(p []ObjectPart) (*Checksum, error) {
|
||||
if !c.CanMergeCRC() {
|
||||
return nil, errors.New("cannot merge this checksum type")
|
||||
}
|
||||
c = c.Base()
|
||||
sort.Slice(p, func(i, j int) bool {
|
||||
return p[i].PartNumber < p[j].PartNumber
|
||||
})
|
||||
|
||||
switch len(p) {
|
||||
case 0:
|
||||
return nil, errors.New("no parts given")
|
||||
case 1:
|
||||
check, err := p[0].ChecksumRaw(c)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &Checksum{
|
||||
Type: c,
|
||||
r: check,
|
||||
}, nil
|
||||
}
|
||||
var merged uint32
|
||||
var merged64 uint64
|
||||
first, err := p[0].ChecksumRaw(c)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
sz := p[0].Size
|
||||
switch c {
|
||||
case ChecksumCRC32, ChecksumCRC32C:
|
||||
merged = binary.BigEndian.Uint32(first)
|
||||
case ChecksumCRC64NVME:
|
||||
merged64 = binary.BigEndian.Uint64(first)
|
||||
}
|
||||
|
||||
poly32 := uint32(crc32.IEEE)
|
||||
if c.Is(ChecksumCRC32C) {
|
||||
poly32 = crc32.Castagnoli
|
||||
}
|
||||
for _, part := range p[1:] {
|
||||
if part.Size == 0 {
|
||||
continue
|
||||
}
|
||||
sz += part.Size
|
||||
pCrc, err := part.ChecksumRaw(c)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
switch c {
|
||||
case ChecksumCRC32, ChecksumCRC32C:
|
||||
merged = crc32Combine(poly32, merged, binary.BigEndian.Uint32(pCrc), part.Size)
|
||||
case ChecksumCRC64NVME:
|
||||
merged64 = crc64Combine(bits.Reverse64(crc64NVMEPolynomial), merged64, binary.BigEndian.Uint64(pCrc), part.Size)
|
||||
}
|
||||
}
|
||||
var tmp [8]byte
|
||||
switch c {
|
||||
case ChecksumCRC32, ChecksumCRC32C:
|
||||
binary.BigEndian.PutUint32(tmp[:], merged)
|
||||
return &Checksum{
|
||||
Type: c,
|
||||
r: tmp[:4],
|
||||
}, nil
|
||||
case ChecksumCRC64NVME:
|
||||
binary.BigEndian.PutUint64(tmp[:], merged64)
|
||||
return &Checksum{
|
||||
Type: c,
|
||||
r: tmp[:8],
|
||||
}, nil
|
||||
default:
|
||||
return nil, errors.New("unknown checksum type")
|
||||
}
|
||||
}
|
||||
|
||||
func addAutoChecksumHeaders(opts *PutObjectOptions) {
|
||||
if opts.UserMetadata == nil {
|
||||
opts.UserMetadata = make(map[string]string, 1)
|
||||
}
|
||||
opts.UserMetadata["X-Amz-Checksum-Algorithm"] = opts.AutoChecksum.String()
|
||||
if opts.AutoChecksum.FullObjectRequested() {
|
||||
opts.UserMetadata["X-Amz-Checksum-Type"] = "FULL_OBJECT"
|
||||
}
|
||||
}
|
||||
|
||||
func applyAutoChecksum(opts *PutObjectOptions, allParts []ObjectPart) {
|
||||
if !opts.AutoChecksum.IsSet() {
|
||||
return
|
||||
}
|
||||
if opts.AutoChecksum.CanComposite() && !opts.AutoChecksum.Is(ChecksumFullObject) {
|
||||
// Add composite hash of hashes.
|
||||
crc, err := opts.AutoChecksum.CompositeChecksum(allParts)
|
||||
if err == nil {
|
||||
opts.UserMetadata = map[string]string{opts.AutoChecksum.Key(): crc.Encoded()}
|
||||
}
|
||||
} else if opts.AutoChecksum.CanMergeCRC() {
|
||||
crc, err := opts.AutoChecksum.FullObjectChecksum(allParts)
|
||||
if err == nil {
|
||||
opts.UserMetadata = map[string]string{opts.AutoChecksum.KeyCapitalized(): crc.Encoded(), "X-Amz-Checksum-Type": "FULL_OBJECT"}
|
||||
}
|
||||
}
|
||||
}
|
||||
+80
@@ -0,0 +1,80 @@
|
||||
# Contributor Covenant Code of Conduct
|
||||
|
||||
## Our Pledge
|
||||
|
||||
In the interest of fostering an open and welcoming environment, we as
|
||||
contributors and maintainers pledge to making participation in our project and
|
||||
our community a harassment-free experience for everyone, regardless of age, body
|
||||
size, disability, ethnicity, gender identity and expression, level of experience,
|
||||
nationality, personal appearance, race, religion, or sexual identity and
|
||||
orientation.
|
||||
|
||||
## Our Standards
|
||||
|
||||
Examples of behavior that contributes to creating a positive environment
|
||||
include:
|
||||
|
||||
* Using welcoming and inclusive language
|
||||
* Being respectful of differing viewpoints and experiences
|
||||
* Gracefully accepting constructive criticism
|
||||
* Focusing on what is best for the community
|
||||
* Showing empathy towards other community members
|
||||
|
||||
Examples of unacceptable behavior by participants include:
|
||||
|
||||
* The use of sexualized language or imagery and unwelcome sexual attention or
|
||||
advances
|
||||
* Trolling, insulting/derogatory comments, and personal or political attacks
|
||||
* Public or private harassment
|
||||
* Publishing others' private information, such as a physical or electronic
|
||||
address, without explicit permission
|
||||
* Other conduct which could reasonably be considered inappropriate in a
|
||||
professional setting
|
||||
|
||||
## Our Responsibilities
|
||||
|
||||
Project maintainers are responsible for clarifying the standards of acceptable
|
||||
behavior and are expected to take appropriate and fair corrective action in
|
||||
response to any instances of unacceptable behavior, in compliance with the
|
||||
licensing terms applying to the Project developments.
|
||||
|
||||
Project maintainers have the right and responsibility to remove, edit, or
|
||||
reject comments, commits, code, wiki edits, issues, and other contributions
|
||||
that are not aligned to this Code of Conduct, or to ban temporarily or
|
||||
permanently any contributor for other behaviors that they deem inappropriate,
|
||||
threatening, offensive, or harmful. However, these actions shall respect the
|
||||
licensing terms of the Project Developments that will always supersede such
|
||||
Code of Conduct.
|
||||
|
||||
## Scope
|
||||
|
||||
This Code of Conduct applies both within project spaces and in public spaces
|
||||
when an individual is representing the project or its community. Examples of
|
||||
representing a project or community include using an official project e-mail
|
||||
address, posting via an official social media account, or acting as an appointed
|
||||
representative at an online or offline event. Representation of a project may be
|
||||
further defined and clarified by project maintainers.
|
||||
|
||||
## Enforcement
|
||||
|
||||
Instances of abusive, harassing, or otherwise unacceptable behavior may be
|
||||
reported by contacting the project team at dev@min.io. The project team
|
||||
will review and investigate all complaints, and will respond in a way that it deems
|
||||
appropriate to the circumstances. The project team is obligated to maintain
|
||||
confidentiality with regard to the reporter of an incident.
|
||||
Further details of specific enforcement policies may be posted separately.
|
||||
|
||||
Project maintainers who do not follow or enforce the Code of Conduct in good
|
||||
faith may face temporary or permanent repercussions as determined by other
|
||||
members of the project's leadership.
|
||||
|
||||
## Attribution
|
||||
|
||||
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
|
||||
available at [http://contributor-covenant.org/version/1/4][version]
|
||||
|
||||
This version includes a clarification to ensure that the code of conduct is in
|
||||
compliance with the free software licensing terms of the project.
|
||||
|
||||
[homepage]: http://contributor-covenant.org
|
||||
[version]: http://contributor-covenant.org/version/1/4/
|
||||
+130
@@ -0,0 +1,130 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2015-2017 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
// Multipart upload defaults.
|
||||
|
||||
// absMinPartSize - absolute minimum part size (5 MiB) below which
|
||||
// a part in a multipart upload may not be uploaded.
|
||||
const absMinPartSize = 1024 * 1024 * 5
|
||||
|
||||
// minPartSize - minimum part size 16MiB per object after which
|
||||
// putObject behaves internally as multipart.
|
||||
const minPartSize = 1024 * 1024 * 16
|
||||
|
||||
// maxPartsCount - maximum number of parts for a single multipart session.
|
||||
const maxPartsCount = 10000
|
||||
|
||||
// maxPartSize - maximum part size 5GiB for a single multipart upload
|
||||
// operation.
|
||||
const maxPartSize = 1024 * 1024 * 1024 * 5
|
||||
|
||||
// maxSinglePutObjectSize - maximum size 5GiB of object per PUT
|
||||
// operation.
|
||||
const maxSinglePutObjectSize = 1024 * 1024 * 1024 * 5
|
||||
|
||||
// maxMultipartPutObjectSize - maximum size 5TiB of object for
|
||||
// Multipart operation.
|
||||
const maxMultipartPutObjectSize = 1024 * 1024 * 1024 * 1024 * 5
|
||||
|
||||
// unsignedPayload - value to be set to X-Amz-Content-Sha256 header when
|
||||
// we don't want to sign the request payload
|
||||
const unsignedPayload = "UNSIGNED-PAYLOAD"
|
||||
|
||||
// unsignedPayloadTrailer value to be set to X-Amz-Content-Sha256 header when
|
||||
// we don't want to sign the request payload, but have a trailer.
|
||||
const unsignedPayloadTrailer = "STREAMING-UNSIGNED-PAYLOAD-TRAILER"
|
||||
|
||||
// Total number of parallel workers used for multipart operation.
|
||||
const totalWorkers = 4
|
||||
|
||||
// Signature related constants.
|
||||
const (
|
||||
signV4Algorithm = "AWS4-HMAC-SHA256"
|
||||
iso8601DateFormat = "20060102T150405Z"
|
||||
)
|
||||
|
||||
const (
|
||||
// GetObjectAttributesTags are tags used to defined
|
||||
// return values for the GetObjectAttributes API
|
||||
GetObjectAttributesTags = "ETag,Checksum,StorageClass,ObjectSize,ObjectParts"
|
||||
// GetObjectAttributesMaxParts defined the default maximum
|
||||
// number of parts returned by GetObjectAttributes
|
||||
GetObjectAttributesMaxParts = 1000
|
||||
)
|
||||
|
||||
const (
|
||||
// Response Headers
|
||||
|
||||
// ETag is a common response header
|
||||
ETag = "ETag"
|
||||
|
||||
// Storage class header.
|
||||
amzStorageClass = "X-Amz-Storage-Class"
|
||||
|
||||
// Website redirect location header
|
||||
amzWebsiteRedirectLocation = "X-Amz-Website-Redirect-Location"
|
||||
|
||||
// GetObjectAttributes headers
|
||||
amzPartNumberMarker = "X-Amz-Part-Number-Marker"
|
||||
amzExpectedBucketOnwer = "X-Amz-Expected-Bucket-Owner"
|
||||
amzMaxParts = "X-Amz-Max-Parts"
|
||||
amzObjectAttributes = "X-Amz-Object-Attributes"
|
||||
|
||||
// Object Tagging headers
|
||||
amzTaggingHeader = "X-Amz-Tagging"
|
||||
amzTaggingHeaderDirective = "X-Amz-Tagging-Directive"
|
||||
|
||||
amzVersionID = "X-Amz-Version-Id"
|
||||
amzTaggingCount = "X-Amz-Tagging-Count"
|
||||
amzExpiration = "X-Amz-Expiration"
|
||||
amzRestore = "X-Amz-Restore"
|
||||
amzReplicationStatus = "X-Amz-Replication-Status"
|
||||
amzDeleteMarker = "X-Amz-Delete-Marker"
|
||||
|
||||
// Object legal hold header
|
||||
amzLegalHoldHeader = "X-Amz-Object-Lock-Legal-Hold"
|
||||
|
||||
// Object retention header
|
||||
amzLockMode = "X-Amz-Object-Lock-Mode"
|
||||
amzLockRetainUntil = "X-Amz-Object-Lock-Retain-Until-Date"
|
||||
amzBypassGovernance = "X-Amz-Bypass-Governance-Retention"
|
||||
|
||||
// Replication status
|
||||
amzBucketReplicationStatus = "X-Amz-Replication-Status"
|
||||
// Minio specific Replication/lifecycle transition extension
|
||||
minIOBucketSourceMTime = "X-Minio-Source-Mtime"
|
||||
|
||||
minIOBucketSourceETag = "X-Minio-Source-Etag"
|
||||
minIOBucketReplicationDeleteMarker = "X-Minio-Source-DeleteMarker"
|
||||
minIOBucketReplicationProxyRequest = "X-Minio-Source-Proxy-Request"
|
||||
minIOBucketReplicationRequest = "X-Minio-Source-Replication-Request"
|
||||
minIOBucketReplicationCheck = "X-Minio-Source-Replication-Check"
|
||||
|
||||
// Header indicates last tag update time on source
|
||||
minIOBucketReplicationTaggingTimestamp = "X-Minio-Source-Replication-Tagging-Timestamp"
|
||||
// Header indicates last retention update time on source
|
||||
minIOBucketReplicationObjectRetentionTimestamp = "X-Minio-Source-Replication-Retention-Timestamp"
|
||||
// Header indicates last legalhold update time on source
|
||||
minIOBucketReplicationObjectLegalHoldTimestamp = "X-Minio-Source-Replication-LegalHold-Timestamp"
|
||||
minIOForceDelete = "x-minio-force-delete"
|
||||
// Header indicates delete marker replication request can be sent by source now.
|
||||
minioTgtReplicationReady = "X-Minio-Replication-Ready"
|
||||
// Header asks if delete marker replication request can be sent by source now.
|
||||
isMinioTgtReplicationReady = "X-Minio-Check-Replication-Ready"
|
||||
)
|
||||
+151
@@ -0,0 +1,151 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2015-2017 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"net/http"
|
||||
|
||||
"github.com/minio/minio-go/v7/pkg/encrypt"
|
||||
)
|
||||
|
||||
// Core - Inherits Client and adds new methods to expose the low level S3 APIs.
|
||||
type Core struct {
|
||||
*Client
|
||||
}
|
||||
|
||||
// NewCore - Returns new initialized a Core client, this CoreClient should be
|
||||
// only used under special conditions such as need to access lower primitives
|
||||
// and being able to use them to write your own wrappers.
|
||||
func NewCore(endpoint string, opts *Options) (*Core, error) {
|
||||
var s3Client Core
|
||||
client, err := New(endpoint, opts)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
s3Client.Client = client
|
||||
return &s3Client, nil
|
||||
}
|
||||
|
||||
// ListObjects - List all the objects at a prefix, optionally with marker and delimiter
|
||||
// you can further filter the results.
|
||||
func (c Core) ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (result ListBucketResult, err error) {
|
||||
return c.listObjectsQuery(context.Background(), bucket, prefix, marker, delimiter, maxKeys, nil)
|
||||
}
|
||||
|
||||
// ListObjectsV2 - Lists all the objects at a prefix, similar to ListObjects() but uses
|
||||
// continuationToken instead of marker to support iteration over the results.
|
||||
func (c Core) ListObjectsV2(bucketName, objectPrefix, startAfter, continuationToken, delimiter string, maxkeys int) (ListBucketV2Result, error) {
|
||||
return c.listObjectsV2Query(context.Background(), bucketName, objectPrefix, continuationToken, true, false, delimiter, startAfter, maxkeys, nil)
|
||||
}
|
||||
|
||||
// CopyObject - copies an object from source object to destination object on server side.
|
||||
func (c Core) CopyObject(ctx context.Context, sourceBucket, sourceObject, destBucket, destObject string, metadata map[string]string, srcOpts CopySrcOptions, dstOpts PutObjectOptions) (ObjectInfo, error) {
|
||||
return c.copyObjectDo(ctx, sourceBucket, sourceObject, destBucket, destObject, metadata, srcOpts, dstOpts)
|
||||
}
|
||||
|
||||
// CopyObjectPart - creates a part in a multipart upload by copying (a
|
||||
// part of) an existing object.
|
||||
func (c Core) CopyObjectPart(ctx context.Context, srcBucket, srcObject, destBucket, destObject, uploadID string,
|
||||
partID int, startOffset, length int64, metadata map[string]string,
|
||||
) (p CompletePart, err error) {
|
||||
return c.copyObjectPartDo(ctx, srcBucket, srcObject, destBucket, destObject, uploadID,
|
||||
partID, startOffset, length, metadata)
|
||||
}
|
||||
|
||||
// PutObject - Upload object. Uploads using single PUT call.
|
||||
func (c Core) PutObject(ctx context.Context, bucket, object string, data io.Reader, size int64, md5Base64, sha256Hex string, opts PutObjectOptions) (UploadInfo, error) {
|
||||
hookReader := newHook(data, opts.Progress)
|
||||
return c.putObjectDo(ctx, bucket, object, hookReader, md5Base64, sha256Hex, size, opts)
|
||||
}
|
||||
|
||||
// NewMultipartUpload - Initiates new multipart upload and returns the new uploadID.
|
||||
func (c Core) NewMultipartUpload(ctx context.Context, bucket, object string, opts PutObjectOptions) (uploadID string, err error) {
|
||||
result, err := c.initiateMultipartUpload(ctx, bucket, object, opts)
|
||||
return result.UploadID, err
|
||||
}
|
||||
|
||||
// ListMultipartUploads - List incomplete uploads.
|
||||
func (c Core) ListMultipartUploads(ctx context.Context, bucket, prefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (result ListMultipartUploadsResult, err error) {
|
||||
return c.listMultipartUploadsQuery(ctx, bucket, keyMarker, uploadIDMarker, prefix, delimiter, maxUploads)
|
||||
}
|
||||
|
||||
// PutObjectPartOptions contains options for PutObjectPart API
|
||||
type PutObjectPartOptions struct {
|
||||
Md5Base64, Sha256Hex string
|
||||
SSE encrypt.ServerSide
|
||||
CustomHeader, Trailer http.Header
|
||||
DisableContentSha256 bool
|
||||
}
|
||||
|
||||
// PutObjectPart - Upload an object part.
|
||||
func (c Core) PutObjectPart(ctx context.Context, bucket, object, uploadID string, partID int,
|
||||
data io.Reader, size int64, opts PutObjectPartOptions,
|
||||
) (ObjectPart, error) {
|
||||
p := uploadPartParams{
|
||||
bucketName: bucket,
|
||||
objectName: object,
|
||||
uploadID: uploadID,
|
||||
reader: data,
|
||||
partNumber: partID,
|
||||
md5Base64: opts.Md5Base64,
|
||||
sha256Hex: opts.Sha256Hex,
|
||||
size: size,
|
||||
sse: opts.SSE,
|
||||
streamSha256: !opts.DisableContentSha256,
|
||||
customHeader: opts.CustomHeader,
|
||||
trailer: opts.Trailer,
|
||||
}
|
||||
return c.uploadPart(ctx, p)
|
||||
}
|
||||
|
||||
// ListObjectParts - List uploaded parts of an incomplete upload.x
|
||||
func (c Core) ListObjectParts(ctx context.Context, bucket, object, uploadID string, partNumberMarker, maxParts int) (result ListObjectPartsResult, err error) {
|
||||
return c.listObjectPartsQuery(ctx, bucket, object, uploadID, partNumberMarker, maxParts)
|
||||
}
|
||||
|
||||
// CompleteMultipartUpload - Concatenate uploaded parts and commit to an object.
|
||||
func (c Core) CompleteMultipartUpload(ctx context.Context, bucket, object, uploadID string, parts []CompletePart, opts PutObjectOptions) (UploadInfo, error) {
|
||||
res, err := c.completeMultipartUpload(ctx, bucket, object, uploadID, completeMultipartUpload{
|
||||
Parts: parts,
|
||||
}, opts)
|
||||
return res, err
|
||||
}
|
||||
|
||||
// AbortMultipartUpload - Abort an incomplete upload.
|
||||
func (c Core) AbortMultipartUpload(ctx context.Context, bucket, object, uploadID string) error {
|
||||
return c.abortMultipartUpload(ctx, bucket, object, uploadID)
|
||||
}
|
||||
|
||||
// GetBucketPolicy - fetches bucket access policy for a given bucket.
|
||||
func (c Core) GetBucketPolicy(ctx context.Context, bucket string) (string, error) {
|
||||
return c.getBucketPolicy(ctx, bucket)
|
||||
}
|
||||
|
||||
// PutBucketPolicy - applies a new bucket access policy for a given bucket.
|
||||
func (c Core) PutBucketPolicy(ctx context.Context, bucket, bucketPolicy string) error {
|
||||
return c.putBucketPolicy(ctx, bucket, bucketPolicy)
|
||||
}
|
||||
|
||||
// GetObject is a lower level API implemented to support reading
|
||||
// partial objects and also downloading objects with special conditions
|
||||
// matching etag, modtime etc.
|
||||
func (c Core) GetObject(ctx context.Context, bucketName, objectName string, opts GetObjectOptions) (io.ReadCloser, ObjectInfo, http.Header, error) {
|
||||
return c.getObject(ctx, bucketName, objectName, opts)
|
||||
}
|
||||
+13912
File diff suppressed because it is too large
Load Diff
+101
@@ -0,0 +1,101 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2015-2017 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package minio
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// hookReader hooks additional reader in the source stream. It is
|
||||
// useful for making progress bars. Second reader is appropriately
|
||||
// notified about the exact number of bytes read from the primary
|
||||
// source on each Read operation.
|
||||
type hookReader struct {
|
||||
mu sync.RWMutex
|
||||
source io.Reader
|
||||
hook io.Reader
|
||||
}
|
||||
|
||||
// Seek implements io.Seeker. Seeks source first, and if necessary
|
||||
// seeks hook if Seek method is appropriately found.
|
||||
func (hr *hookReader) Seek(offset int64, whence int) (n int64, err error) {
|
||||
hr.mu.Lock()
|
||||
defer hr.mu.Unlock()
|
||||
|
||||
// Verify for source has embedded Seeker, use it.
|
||||
sourceSeeker, ok := hr.source.(io.Seeker)
|
||||
if ok {
|
||||
n, err = sourceSeeker.Seek(offset, whence)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
}
|
||||
|
||||
if hr.hook != nil {
|
||||
// Verify if hook has embedded Seeker, use it.
|
||||
hookSeeker, ok := hr.hook.(io.Seeker)
|
||||
if ok {
|
||||
var m int64
|
||||
m, err = hookSeeker.Seek(offset, whence)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if n != m {
|
||||
return 0, fmt.Errorf("hook seeker seeked %d bytes, expected source %d bytes", m, n)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return n, nil
|
||||
}
|
||||
|
||||
// Read implements io.Reader. Always reads from the source, the return
|
||||
// value 'n' number of bytes are reported through the hook. Returns
|
||||
// error for all non io.EOF conditions.
|
||||
func (hr *hookReader) Read(b []byte) (n int, err error) {
|
||||
hr.mu.RLock()
|
||||
defer hr.mu.RUnlock()
|
||||
|
||||
n, err = hr.source.Read(b)
|
||||
if err != nil && err != io.EOF {
|
||||
return n, err
|
||||
}
|
||||
if hr.hook != nil {
|
||||
// Progress the hook with the total read bytes from the source.
|
||||
if _, herr := hr.hook.Read(b[:n]); herr != nil {
|
||||
if herr != io.EOF {
|
||||
return n, herr
|
||||
}
|
||||
}
|
||||
}
|
||||
return n, err
|
||||
}
|
||||
|
||||
// newHook returns a io.ReadSeeker which implements hookReader that
|
||||
// reports the data read from the source to the hook.
|
||||
func newHook(source, hook io.Reader) io.Reader {
|
||||
if hook == nil {
|
||||
return &hookReader{source: source}
|
||||
}
|
||||
return &hookReader{
|
||||
source: source,
|
||||
hook: hook,
|
||||
}
|
||||
}
|
||||
+91
@@ -0,0 +1,91 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2015-2024 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package cors
|
||||
|
||||
import (
|
||||
"encoding/xml"
|
||||
"fmt"
|
||||
"io"
|
||||
"strings"
|
||||
|
||||
"github.com/dustin/go-humanize"
|
||||
)
|
||||
|
||||
const defaultXMLNS = "http://s3.amazonaws.com/doc/2006-03-01/"
|
||||
|
||||
// Config is the container for a CORS configuration for a bucket.
|
||||
type Config struct {
|
||||
XMLNS string `xml:"xmlns,attr,omitempty"`
|
||||
XMLName xml.Name `xml:"CORSConfiguration"`
|
||||
CORSRules []Rule `xml:"CORSRule"`
|
||||
}
|
||||
|
||||
// Rule is a single rule in a CORS configuration.
|
||||
type Rule struct {
|
||||
AllowedHeader []string `xml:"AllowedHeader,omitempty"`
|
||||
AllowedMethod []string `xml:"AllowedMethod,omitempty"`
|
||||
AllowedOrigin []string `xml:"AllowedOrigin,omitempty"`
|
||||
ExposeHeader []string `xml:"ExposeHeader,omitempty"`
|
||||
ID string `xml:"ID,omitempty"`
|
||||
MaxAgeSeconds int `xml:"MaxAgeSeconds,omitempty"`
|
||||
}
|
||||
|
||||
// NewConfig creates a new CORS configuration with the given rules.
|
||||
func NewConfig(rules []Rule) *Config {
|
||||
return &Config{
|
||||
XMLNS: defaultXMLNS,
|
||||
XMLName: xml.Name{
|
||||
Local: "CORSConfiguration",
|
||||
Space: defaultXMLNS,
|
||||
},
|
||||
CORSRules: rules,
|
||||
}
|
||||
}
|
||||
|
||||
// ParseBucketCorsConfig parses a CORS configuration in XML from an io.Reader.
|
||||
func ParseBucketCorsConfig(reader io.Reader) (*Config, error) {
|
||||
var c Config
|
||||
|
||||
// Max size of cors document is 64KiB according to https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketCors.html
|
||||
// This limiter is just for safety so has a max of 128KiB
|
||||
err := xml.NewDecoder(io.LimitReader(reader, 128*humanize.KiByte)).Decode(&c)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("decoding xml: %w", err)
|
||||
}
|
||||
if c.XMLNS == "" {
|
||||
c.XMLNS = defaultXMLNS
|
||||
}
|
||||
for i, rule := range c.CORSRules {
|
||||
for j, method := range rule.AllowedMethod {
|
||||
c.CORSRules[i].AllowedMethod[j] = strings.ToUpper(method)
|
||||
}
|
||||
}
|
||||
return &c, nil
|
||||
}
|
||||
|
||||
// ToXML marshals the CORS configuration to XML.
|
||||
func (c Config) ToXML() ([]byte, error) {
|
||||
if c.XMLNS == "" {
|
||||
c.XMLNS = defaultXMLNS
|
||||
}
|
||||
data, err := xml.Marshal(&c)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("marshaling xml: %w", err)
|
||||
}
|
||||
return append([]byte(xml.Header), data...), nil
|
||||
}
|
||||
Generated
Vendored
+264
@@ -0,0 +1,264 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2020 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package credentials
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/xml"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/minio/minio-go/v7/pkg/signer"
|
||||
)
|
||||
|
||||
// AssumeRoleResponse contains the result of successful AssumeRole request.
|
||||
type AssumeRoleResponse struct {
|
||||
XMLName xml.Name `xml:"https://sts.amazonaws.com/doc/2011-06-15/ AssumeRoleResponse" json:"-"`
|
||||
|
||||
Result AssumeRoleResult `xml:"AssumeRoleResult"`
|
||||
ResponseMetadata struct {
|
||||
RequestID string `xml:"RequestId,omitempty"`
|
||||
} `xml:"ResponseMetadata,omitempty"`
|
||||
}
|
||||
|
||||
// AssumeRoleResult - Contains the response to a successful AssumeRole
|
||||
// request, including temporary credentials that can be used to make
|
||||
// MinIO API requests.
|
||||
type AssumeRoleResult struct {
|
||||
// The identifiers for the temporary security credentials that the operation
|
||||
// returns.
|
||||
AssumedRoleUser AssumedRoleUser `xml:",omitempty"`
|
||||
|
||||
// The temporary security credentials, which include an access key ID, a secret
|
||||
// access key, and a security (or session) token.
|
||||
//
|
||||
// Note: The size of the security token that STS APIs return is not fixed. We
|
||||
// strongly recommend that you make no assumptions about the maximum size. As
|
||||
// of this writing, the typical size is less than 4096 bytes, but that can vary.
|
||||
// Also, future updates to AWS might require larger sizes.
|
||||
Credentials struct {
|
||||
AccessKey string `xml:"AccessKeyId" json:"accessKey,omitempty"`
|
||||
SecretKey string `xml:"SecretAccessKey" json:"secretKey,omitempty"`
|
||||
Expiration time.Time `xml:"Expiration" json:"expiration,omitempty"`
|
||||
SessionToken string `xml:"SessionToken" json:"sessionToken,omitempty"`
|
||||
} `xml:",omitempty"`
|
||||
|
||||
// A percentage value that indicates the size of the policy in packed form.
|
||||
// The service rejects any policy with a packed size greater than 100 percent,
|
||||
// which means the policy exceeded the allowed space.
|
||||
PackedPolicySize int `xml:",omitempty"`
|
||||
}
|
||||
|
||||
// A STSAssumeRole retrieves credentials from MinIO service, and keeps track if
|
||||
// those credentials are expired.
|
||||
type STSAssumeRole struct {
|
||||
Expiry
|
||||
|
||||
// Optional http Client to use when connecting to MinIO STS service
|
||||
// (overrides default client in CredContext)
|
||||
Client *http.Client
|
||||
|
||||
// STS endpoint to fetch STS credentials.
|
||||
STSEndpoint string
|
||||
|
||||
// various options for this request.
|
||||
Options STSAssumeRoleOptions
|
||||
}
|
||||
|
||||
// STSAssumeRoleOptions collection of various input options
|
||||
// to obtain AssumeRole credentials.
|
||||
type STSAssumeRoleOptions struct {
|
||||
// Mandatory inputs.
|
||||
AccessKey string
|
||||
SecretKey string
|
||||
|
||||
SessionToken string // Optional if the first request is made with temporary credentials.
|
||||
Policy string // Optional to assign a policy to the assumed role
|
||||
|
||||
Location string // Optional commonly needed with AWS STS.
|
||||
DurationSeconds int // Optional defaults to 1 hour.
|
||||
|
||||
// Optional only valid if using with AWS STS
|
||||
RoleARN string
|
||||
RoleSessionName string
|
||||
ExternalID string
|
||||
}
|
||||
|
||||
// NewSTSAssumeRole returns a pointer to a new
|
||||
// Credentials object wrapping the STSAssumeRole.
|
||||
func NewSTSAssumeRole(stsEndpoint string, opts STSAssumeRoleOptions) (*Credentials, error) {
|
||||
if opts.AccessKey == "" || opts.SecretKey == "" {
|
||||
return nil, errors.New("AssumeRole credentials access/secretkey is mandatory")
|
||||
}
|
||||
return New(&STSAssumeRole{
|
||||
STSEndpoint: stsEndpoint,
|
||||
Options: opts,
|
||||
}), nil
|
||||
}
|
||||
|
||||
const defaultDurationSeconds = 3600
|
||||
|
||||
// closeResponse close non nil response with any response Body.
|
||||
// convenient wrapper to drain any remaining data on response body.
|
||||
//
|
||||
// Subsequently this allows golang http RoundTripper
|
||||
// to re-use the same connection for future requests.
|
||||
func closeResponse(resp *http.Response) {
|
||||
// Callers should close resp.Body when done reading from it.
|
||||
// If resp.Body is not closed, the Client's underlying RoundTripper
|
||||
// (typically Transport) may not be able to re-use a persistent TCP
|
||||
// connection to the server for a subsequent "keep-alive" request.
|
||||
if resp != nil && resp.Body != nil {
|
||||
// Drain any remaining Body and then close the connection.
|
||||
// Without this closing connection would disallow re-using
|
||||
// the same connection for future uses.
|
||||
// - http://stackoverflow.com/a/17961593/4465767
|
||||
io.Copy(io.Discard, resp.Body)
|
||||
resp.Body.Close()
|
||||
}
|
||||
}
|
||||
|
||||
func getAssumeRoleCredentials(clnt *http.Client, endpoint string, opts STSAssumeRoleOptions) (AssumeRoleResponse, error) {
|
||||
v := url.Values{}
|
||||
v.Set("Action", "AssumeRole")
|
||||
v.Set("Version", STSVersion)
|
||||
if opts.RoleARN != "" {
|
||||
v.Set("RoleArn", opts.RoleARN)
|
||||
}
|
||||
if opts.RoleSessionName != "" {
|
||||
v.Set("RoleSessionName", opts.RoleSessionName)
|
||||
}
|
||||
if opts.DurationSeconds > defaultDurationSeconds {
|
||||
v.Set("DurationSeconds", strconv.Itoa(opts.DurationSeconds))
|
||||
} else {
|
||||
v.Set("DurationSeconds", strconv.Itoa(defaultDurationSeconds))
|
||||
}
|
||||
if opts.Policy != "" {
|
||||
v.Set("Policy", opts.Policy)
|
||||
}
|
||||
if opts.ExternalID != "" {
|
||||
v.Set("ExternalId", opts.ExternalID)
|
||||
}
|
||||
|
||||
u, err := url.Parse(endpoint)
|
||||
if err != nil {
|
||||
return AssumeRoleResponse{}, err
|
||||
}
|
||||
u.Path = "/"
|
||||
|
||||
postBody := strings.NewReader(v.Encode())
|
||||
hash := sha256.New()
|
||||
if _, err = io.Copy(hash, postBody); err != nil {
|
||||
return AssumeRoleResponse{}, err
|
||||
}
|
||||
postBody.Seek(0, 0)
|
||||
|
||||
req, err := http.NewRequest(http.MethodPost, u.String(), postBody)
|
||||
if err != nil {
|
||||
return AssumeRoleResponse{}, err
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||
req.Header.Set("X-Amz-Content-Sha256", hex.EncodeToString(hash.Sum(nil)))
|
||||
if opts.SessionToken != "" {
|
||||
req.Header.Set("X-Amz-Security-Token", opts.SessionToken)
|
||||
}
|
||||
req = signer.SignV4STS(*req, opts.AccessKey, opts.SecretKey, opts.Location)
|
||||
|
||||
resp, err := clnt.Do(req)
|
||||
if err != nil {
|
||||
return AssumeRoleResponse{}, err
|
||||
}
|
||||
defer closeResponse(resp)
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
var errResp ErrorResponse
|
||||
buf, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
return AssumeRoleResponse{}, err
|
||||
}
|
||||
_, err = xmlDecodeAndBody(bytes.NewReader(buf), &errResp)
|
||||
if err != nil {
|
||||
var s3Err Error
|
||||
if _, err = xmlDecodeAndBody(bytes.NewReader(buf), &s3Err); err != nil {
|
||||
return AssumeRoleResponse{}, err
|
||||
}
|
||||
errResp.RequestID = s3Err.RequestID
|
||||
errResp.STSError.Code = s3Err.Code
|
||||
errResp.STSError.Message = s3Err.Message
|
||||
}
|
||||
return AssumeRoleResponse{}, errResp
|
||||
}
|
||||
|
||||
a := AssumeRoleResponse{}
|
||||
if _, err = xmlDecodeAndBody(resp.Body, &a); err != nil {
|
||||
return AssumeRoleResponse{}, err
|
||||
}
|
||||
return a, nil
|
||||
}
|
||||
|
||||
// RetrieveWithCredContext retrieves credentials from the MinIO service.
|
||||
// Error will be returned if the request fails, optional cred context.
|
||||
func (m *STSAssumeRole) RetrieveWithCredContext(cc *CredContext) (Value, error) {
|
||||
if cc == nil {
|
||||
cc = defaultCredContext
|
||||
}
|
||||
|
||||
client := m.Client
|
||||
if client == nil {
|
||||
client = cc.Client
|
||||
}
|
||||
if client == nil {
|
||||
client = defaultCredContext.Client
|
||||
}
|
||||
|
||||
stsEndpoint := m.STSEndpoint
|
||||
if stsEndpoint == "" {
|
||||
stsEndpoint = cc.Endpoint
|
||||
}
|
||||
if stsEndpoint == "" {
|
||||
return Value{}, errors.New("STS endpoint unknown")
|
||||
}
|
||||
|
||||
a, err := getAssumeRoleCredentials(client, stsEndpoint, m.Options)
|
||||
if err != nil {
|
||||
return Value{}, err
|
||||
}
|
||||
|
||||
// Expiry window is set to 10secs.
|
||||
m.SetExpiration(a.Result.Credentials.Expiration, DefaultExpiryWindow)
|
||||
|
||||
return Value{
|
||||
AccessKeyID: a.Result.Credentials.AccessKey,
|
||||
SecretAccessKey: a.Result.Credentials.SecretKey,
|
||||
SessionToken: a.Result.Credentials.SessionToken,
|
||||
Expiration: a.Result.Credentials.Expiration,
|
||||
SignerType: SignatureV4,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Retrieve retrieves credentials from the MinIO service.
|
||||
// Error will be returned if the request fails.
|
||||
func (m *STSAssumeRole) Retrieve() (Value, error) {
|
||||
return m.RetrieveWithCredContext(nil)
|
||||
}
|
||||
+106
@@ -0,0 +1,106 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2017 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package credentials
|
||||
|
||||
// A Chain will search for a provider which returns credentials
|
||||
// and cache that provider until Retrieve is called again.
|
||||
//
|
||||
// The Chain provides a way of chaining multiple providers together
|
||||
// which will pick the first available using priority order of the
|
||||
// Providers in the list.
|
||||
//
|
||||
// If none of the Providers retrieve valid credentials Value, ChainProvider's
|
||||
// Retrieve() will return the no credentials value.
|
||||
//
|
||||
// If a Provider is found which returns valid credentials Value ChainProvider
|
||||
// will cache that Provider for all calls to IsExpired(), until Retrieve is
|
||||
// called again after IsExpired() is true.
|
||||
//
|
||||
// creds := credentials.NewChainCredentials(
|
||||
// []credentials.Provider{
|
||||
// &credentials.EnvAWSS3{},
|
||||
// &credentials.EnvMinio{},
|
||||
// })
|
||||
//
|
||||
// // Usage of ChainCredentials.
|
||||
// mc, err := minio.NewWithCredentials(endpoint, creds, secure, "us-east-1")
|
||||
// if err != nil {
|
||||
// log.Fatalln(err)
|
||||
// }
|
||||
type Chain struct {
|
||||
Providers []Provider
|
||||
curr Provider
|
||||
}
|
||||
|
||||
// NewChainCredentials returns a pointer to a new Credentials object
|
||||
// wrapping a chain of providers.
|
||||
func NewChainCredentials(providers []Provider) *Credentials {
|
||||
return New(&Chain{
|
||||
Providers: append([]Provider{}, providers...),
|
||||
})
|
||||
}
|
||||
|
||||
// RetrieveWithCredContext is like Retrieve with CredContext
|
||||
func (c *Chain) RetrieveWithCredContext(cc *CredContext) (Value, error) {
|
||||
for _, p := range c.Providers {
|
||||
creds, _ := p.RetrieveWithCredContext(cc)
|
||||
// Always prioritize non-anonymous providers, if any.
|
||||
if creds.AccessKeyID == "" && creds.SecretAccessKey == "" {
|
||||
continue
|
||||
}
|
||||
c.curr = p
|
||||
return creds, nil
|
||||
}
|
||||
// At this point we have exhausted all the providers and
|
||||
// are left without any credentials return anonymous.
|
||||
return Value{
|
||||
SignerType: SignatureAnonymous,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Retrieve returns the credentials value, returns no credentials(anonymous)
|
||||
// if no credentials provider returned any value.
|
||||
//
|
||||
// If a provider is found with credentials, it will be cached and any calls
|
||||
// to IsExpired() will return the expired state of the cached provider.
|
||||
func (c *Chain) Retrieve() (Value, error) {
|
||||
for _, p := range c.Providers {
|
||||
creds, _ := p.Retrieve()
|
||||
// Always prioritize non-anonymous providers, if any.
|
||||
if creds.AccessKeyID == "" && creds.SecretAccessKey == "" {
|
||||
continue
|
||||
}
|
||||
c.curr = p
|
||||
return creds, nil
|
||||
}
|
||||
// At this point we have exhausted all the providers and
|
||||
// are left without any credentials return anonymous.
|
||||
return Value{
|
||||
SignerType: SignatureAnonymous,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// IsExpired will returned the expired state of the currently cached provider
|
||||
// if there is one. If there is no current provider, true will be returned.
|
||||
func (c *Chain) IsExpired() bool {
|
||||
if c.curr != nil {
|
||||
return c.curr.IsExpired()
|
||||
}
|
||||
|
||||
return true
|
||||
}
|
||||
Generated
Vendored
+17
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"version": "8",
|
||||
"hosts": {
|
||||
"play": {
|
||||
"url": "https://play.min.io",
|
||||
"accessKey": "Q3AM3UQ867SPQQA43P2F",
|
||||
"secretKey": "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG",
|
||||
"api": "S3v2"
|
||||
},
|
||||
"s3": {
|
||||
"url": "https://s3.amazonaws.com",
|
||||
"accessKey": "accessKey",
|
||||
"secretKey": "secret",
|
||||
"api": "S3v4"
|
||||
}
|
||||
}
|
||||
}
|
||||
Generated
Vendored
+242
@@ -0,0 +1,242 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2017 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package credentials
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
const (
|
||||
// STSVersion sts version string
|
||||
STSVersion = "2011-06-15"
|
||||
|
||||
// How much duration to slash from the given expiration duration
|
||||
defaultExpiryWindow = 0.8
|
||||
)
|
||||
|
||||
// defaultCredContext is used when the credential context doesn't
|
||||
// actually matter or the default context is suitable.
|
||||
var defaultCredContext = &CredContext{Client: http.DefaultClient}
|
||||
|
||||
// A Value is the S3 credentials value for individual credential fields.
|
||||
type Value struct {
|
||||
// S3 Access key ID
|
||||
AccessKeyID string
|
||||
|
||||
// S3 Secret Access Key
|
||||
SecretAccessKey string
|
||||
|
||||
// S3 Session Token
|
||||
SessionToken string
|
||||
|
||||
// Expiration of this credentials - null means no expiration associated
|
||||
Expiration time.Time
|
||||
|
||||
// Signature Type.
|
||||
SignerType SignatureType
|
||||
}
|
||||
|
||||
// A Provider is the interface for any component which will provide credentials
|
||||
// Value. A provider is required to manage its own Expired state, and what to
|
||||
// be expired means.
|
||||
type Provider interface {
|
||||
// RetrieveWithCredContext returns nil if it successfully retrieved the
|
||||
// value. Error is returned if the value were not obtainable, or empty.
|
||||
// optionally takes CredContext for additional context to retrieve credentials.
|
||||
RetrieveWithCredContext(cc *CredContext) (Value, error)
|
||||
|
||||
// Retrieve returns nil if it successfully retrieved the value.
|
||||
// Error is returned if the value were not obtainable, or empty.
|
||||
//
|
||||
// Deprecated: Retrieve() exists for historical compatibility and should not
|
||||
// be used. To get new credentials use the RetrieveWithCredContext function
|
||||
// to ensure the proper context (i.e. HTTP client) will be used.
|
||||
Retrieve() (Value, error)
|
||||
|
||||
// IsExpired returns if the credentials are no longer valid, and need
|
||||
// to be retrieved.
|
||||
IsExpired() bool
|
||||
}
|
||||
|
||||
// CredContext is passed to the Retrieve function of a provider to provide
|
||||
// some additional context to retrieve credentials.
|
||||
type CredContext struct {
|
||||
// Client specifies the HTTP client that should be used if an HTTP
|
||||
// request is to be made to fetch the credentials.
|
||||
Client *http.Client
|
||||
|
||||
// Endpoint specifies the MinIO endpoint that will be used if no
|
||||
// explicit endpoint is provided.
|
||||
Endpoint string
|
||||
}
|
||||
|
||||
// A Expiry provides shared expiration logic to be used by credentials
|
||||
// providers to implement expiry functionality.
|
||||
//
|
||||
// The best method to use this struct is as an anonymous field within the
|
||||
// provider's struct.
|
||||
//
|
||||
// Example:
|
||||
//
|
||||
// type IAMCredentialProvider struct {
|
||||
// Expiry
|
||||
// ...
|
||||
// }
|
||||
type Expiry struct {
|
||||
// The date/time when to expire on
|
||||
expiration time.Time
|
||||
|
||||
// If set will be used by IsExpired to determine the current time.
|
||||
// Defaults to time.Now if CurrentTime is not set.
|
||||
CurrentTime func() time.Time
|
||||
}
|
||||
|
||||
// SetExpiration sets the expiration IsExpired will check when called.
|
||||
//
|
||||
// If window is greater than 0 the expiration time will be reduced by the
|
||||
// window value.
|
||||
//
|
||||
// Using a window is helpful to trigger credentials to expire sooner than
|
||||
// the expiration time given to ensure no requests are made with expired
|
||||
// tokens.
|
||||
func (e *Expiry) SetExpiration(expiration time.Time, window time.Duration) {
|
||||
if e.CurrentTime == nil {
|
||||
e.CurrentTime = time.Now
|
||||
}
|
||||
cut := window
|
||||
if cut < 0 {
|
||||
expireIn := expiration.Sub(e.CurrentTime())
|
||||
cut = time.Duration(float64(expireIn) * (1 - defaultExpiryWindow))
|
||||
}
|
||||
e.expiration = expiration.Add(-cut)
|
||||
}
|
||||
|
||||
// IsExpired returns if the credentials are expired.
|
||||
func (e *Expiry) IsExpired() bool {
|
||||
if e.CurrentTime == nil {
|
||||
e.CurrentTime = time.Now
|
||||
}
|
||||
return e.expiration.Before(e.CurrentTime())
|
||||
}
|
||||
|
||||
// Credentials - A container for synchronous safe retrieval of credentials Value.
|
||||
// Credentials will cache the credentials value until they expire. Once the value
|
||||
// expires the next Get will attempt to retrieve valid credentials.
|
||||
//
|
||||
// Credentials is safe to use across multiple goroutines and will manage the
|
||||
// synchronous state so the Providers do not need to implement their own
|
||||
// synchronization.
|
||||
//
|
||||
// The first Credentials.Get() will always call Provider.Retrieve() to get the
|
||||
// first instance of the credentials Value. All calls to Get() after that
|
||||
// will return the cached credentials Value until IsExpired() returns true.
|
||||
type Credentials struct {
|
||||
sync.Mutex
|
||||
|
||||
creds Value
|
||||
forceRefresh bool
|
||||
provider Provider
|
||||
}
|
||||
|
||||
// New returns a pointer to a new Credentials with the provider set.
|
||||
func New(provider Provider) *Credentials {
|
||||
return &Credentials{
|
||||
provider: provider,
|
||||
forceRefresh: true,
|
||||
}
|
||||
}
|
||||
|
||||
// Get returns the credentials value, or error if the credentials Value failed
|
||||
// to be retrieved.
|
||||
//
|
||||
// Will return the cached credentials Value if it has not expired. If the
|
||||
// credentials Value has expired the Provider's Retrieve() will be called
|
||||
// to refresh the credentials.
|
||||
//
|
||||
// If Credentials.Expire() was called the credentials Value will be force
|
||||
// expired, and the next call to Get() will cause them to be refreshed.
|
||||
//
|
||||
// Deprecated: Get() exists for historical compatibility and should not be
|
||||
// used. To get new credentials use the Credentials.GetWithContext function
|
||||
// to ensure the proper context (i.e. HTTP client) will be used.
|
||||
func (c *Credentials) Get() (Value, error) {
|
||||
return c.GetWithContext(nil)
|
||||
}
|
||||
|
||||
// GetWithContext returns the credentials value, or error if the
|
||||
// credentials Value failed to be retrieved.
|
||||
//
|
||||
// Will return the cached credentials Value if it has not expired. If the
|
||||
// credentials Value has expired the Provider's Retrieve() will be called
|
||||
// to refresh the credentials.
|
||||
//
|
||||
// If Credentials.Expire() was called the credentials Value will be force
|
||||
// expired, and the next call to Get() will cause them to be refreshed.
|
||||
func (c *Credentials) GetWithContext(cc *CredContext) (Value, error) {
|
||||
if c == nil {
|
||||
return Value{}, nil
|
||||
}
|
||||
if cc == nil {
|
||||
cc = defaultCredContext
|
||||
}
|
||||
|
||||
c.Lock()
|
||||
defer c.Unlock()
|
||||
|
||||
if c.isExpired() {
|
||||
creds, err := c.provider.RetrieveWithCredContext(cc)
|
||||
if err != nil {
|
||||
return Value{}, err
|
||||
}
|
||||
c.creds = creds
|
||||
c.forceRefresh = false
|
||||
}
|
||||
|
||||
return c.creds, nil
|
||||
}
|
||||
|
||||
// Expire expires the credentials and forces them to be retrieved on the
|
||||
// next call to Get().
|
||||
//
|
||||
// This will override the Provider's expired state, and force Credentials
|
||||
// to call the Provider's Retrieve().
|
||||
func (c *Credentials) Expire() {
|
||||
c.Lock()
|
||||
defer c.Unlock()
|
||||
|
||||
c.forceRefresh = true
|
||||
}
|
||||
|
||||
// IsExpired returns if the credentials are no longer valid, and need
|
||||
// to be refreshed.
|
||||
//
|
||||
// If the Credentials were forced to be expired with Expire() this will
|
||||
// reflect that override.
|
||||
func (c *Credentials) IsExpired() bool {
|
||||
c.Lock()
|
||||
defer c.Unlock()
|
||||
|
||||
return c.isExpired()
|
||||
}
|
||||
|
||||
// isExpired helper method wrapping the definition of expired credentials.
|
||||
func (c *Credentials) isExpired() bool {
|
||||
return c.forceRefresh || c.provider.IsExpired()
|
||||
}
|
||||
Generated
Vendored
+7
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"Version": 1,
|
||||
"SessionToken": "token",
|
||||
"AccessKeyId": "accessKey",
|
||||
"SecretAccessKey": "secret",
|
||||
"Expiration": "9999-04-27T16:02:25.000Z"
|
||||
}
|
||||
Generated
Vendored
+15
@@ -0,0 +1,15 @@
|
||||
[default]
|
||||
aws_access_key_id = accessKey
|
||||
aws_secret_access_key = secret
|
||||
aws_session_token = token
|
||||
|
||||
[no_token]
|
||||
aws_access_key_id = accessKey
|
||||
aws_secret_access_key = secret
|
||||
|
||||
[with_colon]
|
||||
aws_access_key_id: accessKey
|
||||
aws_secret_access_key: secret
|
||||
|
||||
[with_process]
|
||||
credential_process = /bin/cat credentials.json
|
||||
+60
@@ -0,0 +1,60 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2017 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
// Package credentials provides credential retrieval and management
|
||||
// for S3 compatible object storage.
|
||||
//
|
||||
// By default the Credentials.Get() will cache the successful result of a
|
||||
// Provider's Retrieve() until Provider.IsExpired() returns true. At which
|
||||
// point Credentials will call Provider's Retrieve() to get new credential Value.
|
||||
//
|
||||
// The Provider is responsible for determining when credentials have expired.
|
||||
// It is also important to note that Credentials will always call Retrieve the
|
||||
// first time Credentials.Get() is called.
|
||||
//
|
||||
// Example of using the environment variable credentials.
|
||||
//
|
||||
// creds := NewFromEnv()
|
||||
// // Retrieve the credentials value
|
||||
// credValue, err := creds.Get()
|
||||
// if err != nil {
|
||||
// // handle error
|
||||
// }
|
||||
//
|
||||
// Example of forcing credentials to expire and be refreshed on the next Get().
|
||||
// This may be helpful to proactively expire credentials and refresh them sooner
|
||||
// than they would naturally expire on their own.
|
||||
//
|
||||
// creds := NewFromIAM("")
|
||||
// creds.Expire()
|
||||
// credsValue, err := creds.Get()
|
||||
// // New credentials will be retrieved instead of from cache.
|
||||
//
|
||||
// # Custom Provider
|
||||
//
|
||||
// Each Provider built into this package also provides a helper method to generate
|
||||
// a Credentials pointer setup with the provider. To use a custom Provider just
|
||||
// create a type which satisfies the Provider interface and pass it to the
|
||||
// NewCredentials method.
|
||||
//
|
||||
// type MyProvider struct{}
|
||||
// func (m *MyProvider) Retrieve() (Value, error) {...}
|
||||
// func (m *MyProvider) IsExpired() bool {...}
|
||||
//
|
||||
// creds := NewCredentials(&MyProvider{})
|
||||
// credValue, err := creds.Get()
|
||||
package credentials
|
||||
+80
@@ -0,0 +1,80 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2017 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package credentials
|
||||
|
||||
import "os"
|
||||
|
||||
// A EnvAWS retrieves credentials from the environment variables of the
|
||||
// running process. EnvAWSironment credentials never expire.
|
||||
//
|
||||
// EnvAWSironment variables used:
|
||||
//
|
||||
// * Access Key ID: AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY.
|
||||
// * Secret Access Key: AWS_SECRET_ACCESS_KEY or AWS_SECRET_KEY.
|
||||
// * Secret Token: AWS_SESSION_TOKEN.
|
||||
type EnvAWS struct {
|
||||
retrieved bool
|
||||
}
|
||||
|
||||
// NewEnvAWS returns a pointer to a new Credentials object
|
||||
// wrapping the environment variable provider.
|
||||
func NewEnvAWS() *Credentials {
|
||||
return New(&EnvAWS{})
|
||||
}
|
||||
|
||||
func (e *EnvAWS) retrieve() (Value, error) {
|
||||
e.retrieved = false
|
||||
|
||||
id := os.Getenv("AWS_ACCESS_KEY_ID")
|
||||
if id == "" {
|
||||
id = os.Getenv("AWS_ACCESS_KEY")
|
||||
}
|
||||
|
||||
secret := os.Getenv("AWS_SECRET_ACCESS_KEY")
|
||||
if secret == "" {
|
||||
secret = os.Getenv("AWS_SECRET_KEY")
|
||||
}
|
||||
|
||||
signerType := SignatureV4
|
||||
if id == "" || secret == "" {
|
||||
signerType = SignatureAnonymous
|
||||
}
|
||||
|
||||
e.retrieved = true
|
||||
return Value{
|
||||
AccessKeyID: id,
|
||||
SecretAccessKey: secret,
|
||||
SessionToken: os.Getenv("AWS_SESSION_TOKEN"),
|
||||
SignerType: signerType,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Retrieve retrieves the keys from the environment.
|
||||
func (e *EnvAWS) Retrieve() (Value, error) {
|
||||
return e.retrieve()
|
||||
}
|
||||
|
||||
// RetrieveWithCredContext is like Retrieve (no-op input of Cred Context)
|
||||
func (e *EnvAWS) RetrieveWithCredContext(_ *CredContext) (Value, error) {
|
||||
return e.retrieve()
|
||||
}
|
||||
|
||||
// IsExpired returns if the credentials have been retrieved.
|
||||
func (e *EnvAWS) IsExpired() bool {
|
||||
return !e.retrieved
|
||||
}
|
||||
+77
@@ -0,0 +1,77 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2017 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package credentials
|
||||
|
||||
import "os"
|
||||
|
||||
// A EnvMinio retrieves credentials from the environment variables of the
|
||||
// running process. EnvMinioironment credentials never expire.
|
||||
//
|
||||
// Environment variables used:
|
||||
//
|
||||
// * Access Key ID: MINIO_ACCESS_KEY.
|
||||
// * Secret Access Key: MINIO_SECRET_KEY.
|
||||
// * Access Key ID: MINIO_ROOT_USER.
|
||||
// * Secret Access Key: MINIO_ROOT_PASSWORD.
|
||||
type EnvMinio struct {
|
||||
retrieved bool
|
||||
}
|
||||
|
||||
// NewEnvMinio returns a pointer to a new Credentials object
|
||||
// wrapping the environment variable provider.
|
||||
func NewEnvMinio() *Credentials {
|
||||
return New(&EnvMinio{})
|
||||
}
|
||||
|
||||
func (e *EnvMinio) retrieve() (Value, error) {
|
||||
e.retrieved = false
|
||||
|
||||
id := os.Getenv("MINIO_ROOT_USER")
|
||||
secret := os.Getenv("MINIO_ROOT_PASSWORD")
|
||||
|
||||
signerType := SignatureV4
|
||||
if id == "" || secret == "" {
|
||||
id = os.Getenv("MINIO_ACCESS_KEY")
|
||||
secret = os.Getenv("MINIO_SECRET_KEY")
|
||||
if id == "" || secret == "" {
|
||||
signerType = SignatureAnonymous
|
||||
}
|
||||
}
|
||||
|
||||
e.retrieved = true
|
||||
return Value{
|
||||
AccessKeyID: id,
|
||||
SecretAccessKey: secret,
|
||||
SignerType: signerType,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Retrieve retrieves the keys from the environment.
|
||||
func (e *EnvMinio) Retrieve() (Value, error) {
|
||||
return e.retrieve()
|
||||
}
|
||||
|
||||
// RetrieveWithCredContext is like Retrieve() (no-op input cred context)
|
||||
func (e *EnvMinio) RetrieveWithCredContext(_ *CredContext) (Value, error) {
|
||||
return e.retrieve()
|
||||
}
|
||||
|
||||
// IsExpired returns if the credentials have been retrieved.
|
||||
func (e *EnvMinio) IsExpired() bool {
|
||||
return !e.retrieved
|
||||
}
|
||||
Generated
Vendored
+95
@@ -0,0 +1,95 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2021 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package credentials
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/xml"
|
||||
"fmt"
|
||||
"io"
|
||||
)
|
||||
|
||||
// ErrorResponse - Is the typed error returned.
|
||||
// ErrorResponse struct should be comparable since it is compared inside
|
||||
// golang http API (https://github.com/golang/go/issues/29768)
|
||||
type ErrorResponse struct {
|
||||
XMLName xml.Name `xml:"https://sts.amazonaws.com/doc/2011-06-15/ ErrorResponse" json:"-"`
|
||||
STSError struct {
|
||||
Type string `xml:"Type"`
|
||||
Code string `xml:"Code"`
|
||||
Message string `xml:"Message"`
|
||||
} `xml:"Error"`
|
||||
RequestID string `xml:"RequestId"`
|
||||
}
|
||||
|
||||
// Error - Is the typed error returned by all API operations.
|
||||
type Error struct {
|
||||
XMLName xml.Name `xml:"Error" json:"-"`
|
||||
Code string
|
||||
Message string
|
||||
BucketName string
|
||||
Key string
|
||||
Resource string
|
||||
RequestID string `xml:"RequestId"`
|
||||
HostID string `xml:"HostId"`
|
||||
|
||||
// Region where the bucket is located. This header is returned
|
||||
// only in HEAD bucket and ListObjects response.
|
||||
Region string
|
||||
|
||||
// Captures the server string returned in response header.
|
||||
Server string
|
||||
|
||||
// Underlying HTTP status code for the returned error
|
||||
StatusCode int `xml:"-" json:"-"`
|
||||
}
|
||||
|
||||
// Error - Returns S3 error string.
|
||||
func (e Error) Error() string {
|
||||
if e.Message == "" {
|
||||
return fmt.Sprintf("Error response code %s.", e.Code)
|
||||
}
|
||||
return e.Message
|
||||
}
|
||||
|
||||
// Error - Returns STS error string.
|
||||
func (e ErrorResponse) Error() string {
|
||||
if e.STSError.Message == "" {
|
||||
return fmt.Sprintf("Error response code %s.", e.STSError.Code)
|
||||
}
|
||||
return e.STSError.Message
|
||||
}
|
||||
|
||||
// xmlDecoder provide decoded value in xml.
|
||||
func xmlDecoder(body io.Reader, v interface{}) error {
|
||||
d := xml.NewDecoder(body)
|
||||
return d.Decode(v)
|
||||
}
|
||||
|
||||
// xmlDecodeAndBody reads the whole body up to 1MB and
|
||||
// tries to XML decode it into v.
|
||||
// The body that was read and any error from reading or decoding is returned.
|
||||
func xmlDecodeAndBody(bodyReader io.Reader, v interface{}) ([]byte, error) {
|
||||
// read the whole body (up to 1MB)
|
||||
const maxBodyLength = 1 << 20
|
||||
body, err := io.ReadAll(io.LimitReader(bodyReader, maxBodyLength))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return bytes.TrimSpace(body), xmlDecoder(bytes.NewReader(body), v)
|
||||
}
|
||||
Generated
Vendored
+167
@@ -0,0 +1,167 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2017 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package credentials
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/go-ini/ini"
|
||||
)
|
||||
|
||||
// A externalProcessCredentials stores the output of a credential_process
|
||||
type externalProcessCredentials struct {
|
||||
Version int
|
||||
SessionToken string
|
||||
AccessKeyID string `json:"AccessKeyId"`
|
||||
SecretAccessKey string
|
||||
Expiration time.Time
|
||||
}
|
||||
|
||||
// A FileAWSCredentials retrieves credentials from the current user's home
|
||||
// directory, and keeps track if those credentials are expired.
|
||||
//
|
||||
// Profile ini file example: $HOME/.aws/credentials
|
||||
type FileAWSCredentials struct {
|
||||
Expiry
|
||||
|
||||
// Path to the shared credentials file.
|
||||
//
|
||||
// If empty will look for "AWS_SHARED_CREDENTIALS_FILE" env variable. If the
|
||||
// env value is empty will default to current user's home directory.
|
||||
// Linux/OSX: "$HOME/.aws/credentials"
|
||||
// Windows: "%USERPROFILE%\.aws\credentials"
|
||||
Filename string
|
||||
|
||||
// AWS Profile to extract credentials from the shared credentials file. If empty
|
||||
// will default to environment variable "AWS_PROFILE" or "default" if
|
||||
// environment variable is also not set.
|
||||
Profile string
|
||||
|
||||
// retrieved states if the credentials have been successfully retrieved.
|
||||
retrieved bool
|
||||
}
|
||||
|
||||
// NewFileAWSCredentials returns a pointer to a new Credentials object
|
||||
// wrapping the Profile file provider.
|
||||
func NewFileAWSCredentials(filename, profile string) *Credentials {
|
||||
return New(&FileAWSCredentials{
|
||||
Filename: filename,
|
||||
Profile: profile,
|
||||
})
|
||||
}
|
||||
|
||||
func (p *FileAWSCredentials) retrieve() (Value, error) {
|
||||
if p.Filename == "" {
|
||||
p.Filename = os.Getenv("AWS_SHARED_CREDENTIALS_FILE")
|
||||
if p.Filename == "" {
|
||||
homeDir, err := os.UserHomeDir()
|
||||
if err != nil {
|
||||
return Value{}, err
|
||||
}
|
||||
p.Filename = filepath.Join(homeDir, ".aws", "credentials")
|
||||
}
|
||||
}
|
||||
if p.Profile == "" {
|
||||
p.Profile = os.Getenv("AWS_PROFILE")
|
||||
if p.Profile == "" {
|
||||
p.Profile = "default"
|
||||
}
|
||||
}
|
||||
|
||||
p.retrieved = false
|
||||
|
||||
iniProfile, err := loadProfile(p.Filename, p.Profile)
|
||||
if err != nil {
|
||||
return Value{}, err
|
||||
}
|
||||
|
||||
// Default to empty string if not found.
|
||||
id := iniProfile.Key("aws_access_key_id")
|
||||
// Default to empty string if not found.
|
||||
secret := iniProfile.Key("aws_secret_access_key")
|
||||
// Default to empty string if not found.
|
||||
token := iniProfile.Key("aws_session_token")
|
||||
|
||||
// If credential_process is defined, obtain credentials by executing
|
||||
// the external process
|
||||
credentialProcess := strings.TrimSpace(iniProfile.Key("credential_process").String())
|
||||
if credentialProcess != "" {
|
||||
args := strings.Fields(credentialProcess)
|
||||
if len(args) <= 1 {
|
||||
return Value{}, errors.New("invalid credential process args")
|
||||
}
|
||||
cmd := exec.Command(args[0], args[1:]...)
|
||||
out, err := cmd.Output()
|
||||
if err != nil {
|
||||
return Value{}, err
|
||||
}
|
||||
var externalProcessCredentials externalProcessCredentials
|
||||
err = json.Unmarshal([]byte(out), &externalProcessCredentials)
|
||||
if err != nil {
|
||||
return Value{}, err
|
||||
}
|
||||
p.retrieved = true
|
||||
p.SetExpiration(externalProcessCredentials.Expiration, DefaultExpiryWindow)
|
||||
return Value{
|
||||
AccessKeyID: externalProcessCredentials.AccessKeyID,
|
||||
SecretAccessKey: externalProcessCredentials.SecretAccessKey,
|
||||
SessionToken: externalProcessCredentials.SessionToken,
|
||||
Expiration: externalProcessCredentials.Expiration,
|
||||
SignerType: SignatureV4,
|
||||
}, nil
|
||||
}
|
||||
p.retrieved = true
|
||||
return Value{
|
||||
AccessKeyID: id.String(),
|
||||
SecretAccessKey: secret.String(),
|
||||
SessionToken: token.String(),
|
||||
SignerType: SignatureV4,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Retrieve reads and extracts the shared credentials from the current
|
||||
// users home directory.
|
||||
func (p *FileAWSCredentials) Retrieve() (Value, error) {
|
||||
return p.retrieve()
|
||||
}
|
||||
|
||||
// RetrieveWithCredContext is like Retrieve(), cred context is no-op for File credentials
|
||||
func (p *FileAWSCredentials) RetrieveWithCredContext(_ *CredContext) (Value, error) {
|
||||
return p.retrieve()
|
||||
}
|
||||
|
||||
// loadProfiles loads from the file pointed to by shared credentials filename for profile.
|
||||
// The credentials retrieved from the profile will be returned or error. Error will be
|
||||
// returned if it fails to read from the file, or the data is invalid.
|
||||
func loadProfile(filename, profile string) (*ini.Section, error) {
|
||||
config, err := ini.Load(filename)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
iniProfile, err := config.GetSection(profile)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return iniProfile, nil
|
||||
}
|
||||
Generated
Vendored
+146
@@ -0,0 +1,146 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2017 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package credentials
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
|
||||
"github.com/goccy/go-json"
|
||||
)
|
||||
|
||||
// A FileMinioClient retrieves credentials from the current user's home
|
||||
// directory, and keeps track if those credentials are expired.
|
||||
//
|
||||
// Configuration file example: $HOME/.mc/config.json
|
||||
type FileMinioClient struct {
|
||||
// Path to the shared credentials file.
|
||||
//
|
||||
// If empty will look for "MINIO_SHARED_CREDENTIALS_FILE" env variable. If the
|
||||
// env value is empty will default to current user's home directory.
|
||||
// Linux/OSX: "$HOME/.mc/config.json"
|
||||
// Windows: "%USERALIAS%\mc\config.json"
|
||||
Filename string
|
||||
|
||||
// MinIO Alias to extract credentials from the shared credentials file. If empty
|
||||
// will default to environment variable "MINIO_ALIAS" or "s3" if
|
||||
// environment variable is also not set.
|
||||
Alias string
|
||||
|
||||
// retrieved states if the credentials have been successfully retrieved.
|
||||
retrieved bool
|
||||
}
|
||||
|
||||
// NewFileMinioClient returns a pointer to a new Credentials object
|
||||
// wrapping the Alias file provider.
|
||||
func NewFileMinioClient(filename, alias string) *Credentials {
|
||||
return New(&FileMinioClient{
|
||||
Filename: filename,
|
||||
Alias: alias,
|
||||
})
|
||||
}
|
||||
|
||||
func (p *FileMinioClient) retrieve() (Value, error) {
|
||||
if p.Filename == "" {
|
||||
if value, ok := os.LookupEnv("MINIO_SHARED_CREDENTIALS_FILE"); ok {
|
||||
p.Filename = value
|
||||
} else {
|
||||
homeDir, err := os.UserHomeDir()
|
||||
if err != nil {
|
||||
return Value{}, err
|
||||
}
|
||||
p.Filename = filepath.Join(homeDir, ".mc", "config.json")
|
||||
if runtime.GOOS == "windows" {
|
||||
p.Filename = filepath.Join(homeDir, "mc", "config.json")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if p.Alias == "" {
|
||||
p.Alias = os.Getenv("MINIO_ALIAS")
|
||||
if p.Alias == "" {
|
||||
p.Alias = "s3"
|
||||
}
|
||||
}
|
||||
|
||||
p.retrieved = false
|
||||
|
||||
hostCfg, err := loadAlias(p.Filename, p.Alias)
|
||||
if err != nil {
|
||||
return Value{}, err
|
||||
}
|
||||
|
||||
p.retrieved = true
|
||||
return Value{
|
||||
AccessKeyID: hostCfg.AccessKey,
|
||||
SecretAccessKey: hostCfg.SecretKey,
|
||||
SignerType: parseSignatureType(hostCfg.API),
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Retrieve reads and extracts the shared credentials from the current
|
||||
// users home directory.
|
||||
func (p *FileMinioClient) Retrieve() (Value, error) {
|
||||
return p.retrieve()
|
||||
}
|
||||
|
||||
// RetrieveWithCredContext - is like Retrieve()
|
||||
func (p *FileMinioClient) RetrieveWithCredContext(_ *CredContext) (Value, error) {
|
||||
return p.retrieve()
|
||||
}
|
||||
|
||||
// IsExpired returns if the shared credentials have expired.
|
||||
func (p *FileMinioClient) IsExpired() bool {
|
||||
return !p.retrieved
|
||||
}
|
||||
|
||||
// hostConfig configuration of a host.
|
||||
type hostConfig struct {
|
||||
URL string `json:"url"`
|
||||
AccessKey string `json:"accessKey"`
|
||||
SecretKey string `json:"secretKey"`
|
||||
API string `json:"api"`
|
||||
}
|
||||
|
||||
// config config version.
|
||||
type config struct {
|
||||
Version string `json:"version"`
|
||||
Hosts map[string]hostConfig `json:"hosts"`
|
||||
Aliases map[string]hostConfig `json:"aliases"`
|
||||
}
|
||||
|
||||
// loadAliass loads from the file pointed to by shared credentials filename for alias.
|
||||
// The credentials retrieved from the alias will be returned or error. Error will be
|
||||
// returned if it fails to read from the file.
|
||||
func loadAlias(filename, alias string) (hostConfig, error) {
|
||||
cfg := &config{}
|
||||
configBytes, err := os.ReadFile(filename)
|
||||
if err != nil {
|
||||
return hostConfig{}, err
|
||||
}
|
||||
if err = json.Unmarshal(configBytes, cfg); err != nil {
|
||||
return hostConfig{}, err
|
||||
}
|
||||
|
||||
if cfg.Version == "10" {
|
||||
return cfg.Aliases[alias], nil
|
||||
}
|
||||
|
||||
return cfg.Hosts[alias], nil
|
||||
}
|
||||
+475
@@ -0,0 +1,475 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2017 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package credentials
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"path"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/goccy/go-json"
|
||||
)
|
||||
|
||||
// DefaultExpiryWindow - Default expiry window.
|
||||
// ExpiryWindow will allow the credentials to trigger refreshing
|
||||
// prior to the credentials actually expiring. This is beneficial
|
||||
// so race conditions with expiring credentials do not cause
|
||||
// request to fail unexpectedly due to ExpiredTokenException exceptions.
|
||||
// DefaultExpiryWindow can be used as parameter to (*Expiry).SetExpiration.
|
||||
// When used the tokens refresh will be triggered when 80% of the elapsed
|
||||
// time until the actual expiration time is passed.
|
||||
const DefaultExpiryWindow = -1
|
||||
|
||||
// A IAM retrieves credentials from the EC2 service, and keeps track if
|
||||
// those credentials are expired.
|
||||
type IAM struct {
|
||||
Expiry
|
||||
|
||||
// Optional http Client to use when connecting to IAM metadata service
|
||||
// (overrides default client in CredContext)
|
||||
Client *http.Client
|
||||
|
||||
// Custom endpoint to fetch IAM role credentials.
|
||||
Endpoint string
|
||||
|
||||
// Region configurable custom region for STS
|
||||
Region string
|
||||
|
||||
// Support for container authorization token https://docs.aws.amazon.com/sdkref/latest/guide/feature-container-credentials.html
|
||||
Container struct {
|
||||
AuthorizationToken string
|
||||
AuthorizationTokenFile string
|
||||
CredentialsFullURI string
|
||||
CredentialsRelativeURI string
|
||||
}
|
||||
|
||||
// EKS based k8s RBAC authorization - https://docs.aws.amazon.com/eks/latest/userguide/pod-configuration.html
|
||||
EKSIdentity struct {
|
||||
TokenFile string
|
||||
RoleARN string
|
||||
RoleSessionName string
|
||||
}
|
||||
}
|
||||
|
||||
// IAM Roles for Amazon EC2
|
||||
// http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
|
||||
const (
|
||||
DefaultIAMRoleEndpoint = "http://169.254.169.254"
|
||||
DefaultECSRoleEndpoint = "http://169.254.170.2"
|
||||
DefaultSTSRoleEndpoint = "https://sts.amazonaws.com"
|
||||
DefaultIAMSecurityCredsPath = "/latest/meta-data/iam/security-credentials/"
|
||||
TokenRequestTTLHeader = "X-aws-ec2-metadata-token-ttl-seconds"
|
||||
TokenPath = "/latest/api/token"
|
||||
TokenTTL = "21600"
|
||||
TokenRequestHeader = "X-aws-ec2-metadata-token"
|
||||
)
|
||||
|
||||
// NewIAM returns a pointer to a new Credentials object wrapping the IAM.
|
||||
func NewIAM(endpoint string) *Credentials {
|
||||
return New(&IAM{
|
||||
Endpoint: endpoint,
|
||||
})
|
||||
}
|
||||
|
||||
// RetrieveWithCredContext is like Retrieve with Cred Context
|
||||
func (m *IAM) RetrieveWithCredContext(cc *CredContext) (Value, error) {
|
||||
if cc == nil {
|
||||
cc = defaultCredContext
|
||||
}
|
||||
|
||||
token := os.Getenv("AWS_CONTAINER_AUTHORIZATION_TOKEN")
|
||||
if token == "" {
|
||||
token = m.Container.AuthorizationToken
|
||||
}
|
||||
|
||||
tokenFile := os.Getenv("AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE")
|
||||
if tokenFile == "" {
|
||||
tokenFile = m.Container.AuthorizationToken
|
||||
}
|
||||
|
||||
relativeURI := os.Getenv("AWS_CONTAINER_CREDENTIALS_RELATIVE_URI")
|
||||
if relativeURI == "" {
|
||||
relativeURI = m.Container.CredentialsRelativeURI
|
||||
}
|
||||
|
||||
fullURI := os.Getenv("AWS_CONTAINER_CREDENTIALS_FULL_URI")
|
||||
if fullURI == "" {
|
||||
fullURI = m.Container.CredentialsFullURI
|
||||
}
|
||||
|
||||
identityFile := os.Getenv("AWS_WEB_IDENTITY_TOKEN_FILE")
|
||||
if identityFile == "" {
|
||||
identityFile = m.EKSIdentity.TokenFile
|
||||
}
|
||||
|
||||
roleArn := os.Getenv("AWS_ROLE_ARN")
|
||||
if roleArn == "" {
|
||||
roleArn = m.EKSIdentity.RoleARN
|
||||
}
|
||||
|
||||
roleSessionName := os.Getenv("AWS_ROLE_SESSION_NAME")
|
||||
if roleSessionName == "" {
|
||||
roleSessionName = m.EKSIdentity.RoleSessionName
|
||||
}
|
||||
|
||||
region := os.Getenv("AWS_REGION")
|
||||
if region == "" {
|
||||
region = m.Region
|
||||
}
|
||||
|
||||
var roleCreds ec2RoleCredRespBody
|
||||
var err error
|
||||
|
||||
client := m.Client
|
||||
if client == nil {
|
||||
client = cc.Client
|
||||
}
|
||||
if client == nil {
|
||||
client = defaultCredContext.Client
|
||||
}
|
||||
|
||||
endpoint := m.Endpoint
|
||||
if endpoint == "" {
|
||||
endpoint = cc.Endpoint
|
||||
}
|
||||
|
||||
switch {
|
||||
case identityFile != "":
|
||||
if len(endpoint) == 0 {
|
||||
if region != "" {
|
||||
if strings.HasPrefix(region, "cn-") {
|
||||
endpoint = "https://sts." + region + ".amazonaws.com.cn"
|
||||
} else {
|
||||
endpoint = "https://sts." + region + ".amazonaws.com"
|
||||
}
|
||||
} else {
|
||||
endpoint = DefaultSTSRoleEndpoint
|
||||
}
|
||||
}
|
||||
|
||||
creds := &STSWebIdentity{
|
||||
Client: client,
|
||||
STSEndpoint: endpoint,
|
||||
GetWebIDTokenExpiry: func() (*WebIdentityToken, error) {
|
||||
token, err := os.ReadFile(identityFile)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &WebIdentityToken{Token: string(token)}, nil
|
||||
},
|
||||
RoleARN: roleArn,
|
||||
roleSessionName: roleSessionName,
|
||||
}
|
||||
|
||||
stsWebIdentityCreds, err := creds.RetrieveWithCredContext(cc)
|
||||
if err == nil {
|
||||
m.SetExpiration(creds.Expiration(), DefaultExpiryWindow)
|
||||
}
|
||||
return stsWebIdentityCreds, err
|
||||
|
||||
case relativeURI != "":
|
||||
if len(endpoint) == 0 {
|
||||
endpoint = fmt.Sprintf("%s%s", DefaultECSRoleEndpoint, relativeURI)
|
||||
}
|
||||
|
||||
roleCreds, err = getEcsTaskCredentials(client, endpoint, token)
|
||||
|
||||
case tokenFile != "" && fullURI != "":
|
||||
endpoint = fullURI
|
||||
roleCreds, err = getEKSPodIdentityCredentials(client, endpoint, tokenFile)
|
||||
|
||||
case fullURI != "":
|
||||
if len(endpoint) == 0 {
|
||||
endpoint = fullURI
|
||||
var ok bool
|
||||
if ok, err = isLoopback(endpoint); !ok {
|
||||
if err == nil {
|
||||
err = fmt.Errorf("uri host is not a loopback address: %s", endpoint)
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
roleCreds, err = getEcsTaskCredentials(client, endpoint, token)
|
||||
|
||||
default:
|
||||
roleCreds, err = getCredentials(client, endpoint)
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
return Value{}, err
|
||||
}
|
||||
// Expiry window is set to 10secs.
|
||||
m.SetExpiration(roleCreds.Expiration, DefaultExpiryWindow)
|
||||
|
||||
return Value{
|
||||
AccessKeyID: roleCreds.AccessKeyID,
|
||||
SecretAccessKey: roleCreds.SecretAccessKey,
|
||||
SessionToken: roleCreds.Token,
|
||||
Expiration: roleCreds.Expiration,
|
||||
SignerType: SignatureV4,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Retrieve retrieves credentials from the EC2 service.
|
||||
// Error will be returned if the request fails, or unable to extract
|
||||
// the desired
|
||||
func (m *IAM) Retrieve() (Value, error) {
|
||||
return m.RetrieveWithCredContext(nil)
|
||||
}
|
||||
|
||||
// A ec2RoleCredRespBody provides the shape for unmarshaling credential
|
||||
// request responses.
|
||||
type ec2RoleCredRespBody struct {
|
||||
// Success State
|
||||
Expiration time.Time
|
||||
AccessKeyID string
|
||||
SecretAccessKey string
|
||||
Token string
|
||||
|
||||
// Error state
|
||||
Code string
|
||||
Message string
|
||||
|
||||
// Unused params.
|
||||
LastUpdated time.Time
|
||||
Type string
|
||||
}
|
||||
|
||||
// Get the final IAM role URL where the request will
|
||||
// be sent to fetch the rolling access credentials.
|
||||
// http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
|
||||
func getIAMRoleURL(endpoint string) (*url.URL, error) {
|
||||
u, err := url.Parse(endpoint)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
u.Path = DefaultIAMSecurityCredsPath
|
||||
return u, nil
|
||||
}
|
||||
|
||||
// listRoleNames lists of credential role names associated
|
||||
// with the current EC2 service. If there are no credentials,
|
||||
// or there is an error making or receiving the request.
|
||||
// http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
|
||||
func listRoleNames(client *http.Client, u *url.URL, token string) ([]string, error) {
|
||||
req, err := http.NewRequest(http.MethodGet, u.String(), nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if token != "" {
|
||||
req.Header.Add(TokenRequestHeader, token)
|
||||
}
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return nil, errors.New(resp.Status)
|
||||
}
|
||||
|
||||
credsList := []string{}
|
||||
s := bufio.NewScanner(resp.Body)
|
||||
for s.Scan() {
|
||||
credsList = append(credsList, s.Text())
|
||||
}
|
||||
|
||||
if err := s.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return credsList, nil
|
||||
}
|
||||
|
||||
func getEcsTaskCredentials(client *http.Client, endpoint, token string) (ec2RoleCredRespBody, error) {
|
||||
req, err := http.NewRequest(http.MethodGet, endpoint, nil)
|
||||
if err != nil {
|
||||
return ec2RoleCredRespBody{}, err
|
||||
}
|
||||
|
||||
if token != "" {
|
||||
req.Header.Set("Authorization", token)
|
||||
}
|
||||
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return ec2RoleCredRespBody{}, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return ec2RoleCredRespBody{}, errors.New(resp.Status)
|
||||
}
|
||||
|
||||
respCreds := ec2RoleCredRespBody{}
|
||||
if err := json.NewDecoder(resp.Body).Decode(&respCreds); err != nil {
|
||||
return ec2RoleCredRespBody{}, err
|
||||
}
|
||||
|
||||
return respCreds, nil
|
||||
}
|
||||
|
||||
func getEKSPodIdentityCredentials(client *http.Client, endpoint string, tokenFile string) (ec2RoleCredRespBody, error) {
|
||||
if tokenFile != "" {
|
||||
bytes, err := os.ReadFile(tokenFile)
|
||||
if err != nil {
|
||||
return ec2RoleCredRespBody{}, fmt.Errorf("getEKSPodIdentityCredentials: failed to read token file:%s", err)
|
||||
}
|
||||
token := string(bytes)
|
||||
return getEcsTaskCredentials(client, endpoint, token)
|
||||
}
|
||||
return ec2RoleCredRespBody{}, fmt.Errorf("getEKSPodIdentityCredentials: no tokenFile found")
|
||||
}
|
||||
|
||||
func fetchIMDSToken(client *http.Client, endpoint string) (string, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
|
||||
defer cancel()
|
||||
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPut, endpoint+TokenPath, nil)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
req.Header.Add(TokenRequestTTLHeader, TokenTTL)
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
data, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return "", errors.New(resp.Status)
|
||||
}
|
||||
return string(data), nil
|
||||
}
|
||||
|
||||
// getCredentials - obtains the credentials from the IAM role name associated with
|
||||
// the current EC2 service.
|
||||
//
|
||||
// If the credentials cannot be found, or there is an error
|
||||
// reading the response an error will be returned.
|
||||
func getCredentials(client *http.Client, endpoint string) (ec2RoleCredRespBody, error) {
|
||||
if endpoint == "" {
|
||||
endpoint = DefaultIAMRoleEndpoint
|
||||
}
|
||||
|
||||
// https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html
|
||||
token, err := fetchIMDSToken(client, endpoint)
|
||||
if err != nil {
|
||||
// Return only errors for valid situations, if the IMDSv2 is not enabled
|
||||
// we will not be able to get the token, in such a situation we have
|
||||
// to rely on IMDSv1 behavior as a fallback, this check ensures that.
|
||||
// Refer https://github.com/minio/minio-go/issues/1866
|
||||
if !errors.Is(err, context.DeadlineExceeded) && !errors.Is(err, context.Canceled) {
|
||||
return ec2RoleCredRespBody{}, err
|
||||
}
|
||||
}
|
||||
|
||||
// http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
|
||||
u, err := getIAMRoleURL(endpoint)
|
||||
if err != nil {
|
||||
return ec2RoleCredRespBody{}, err
|
||||
}
|
||||
|
||||
// http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
|
||||
roleNames, err := listRoleNames(client, u, token)
|
||||
if err != nil {
|
||||
return ec2RoleCredRespBody{}, err
|
||||
}
|
||||
|
||||
if len(roleNames) == 0 {
|
||||
return ec2RoleCredRespBody{}, errors.New("No IAM roles attached to this EC2 service")
|
||||
}
|
||||
|
||||
// http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
|
||||
// - An instance profile can contain only one IAM role. This limit cannot be increased.
|
||||
roleName := roleNames[0]
|
||||
|
||||
// http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
|
||||
// The following command retrieves the security credentials for an
|
||||
// IAM role named `s3access`.
|
||||
//
|
||||
// $ curl http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access
|
||||
//
|
||||
u.Path = path.Join(u.Path, roleName)
|
||||
req, err := http.NewRequest(http.MethodGet, u.String(), nil)
|
||||
if err != nil {
|
||||
return ec2RoleCredRespBody{}, err
|
||||
}
|
||||
if token != "" {
|
||||
req.Header.Add(TokenRequestHeader, token)
|
||||
}
|
||||
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return ec2RoleCredRespBody{}, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return ec2RoleCredRespBody{}, errors.New(resp.Status)
|
||||
}
|
||||
|
||||
respCreds := ec2RoleCredRespBody{}
|
||||
if err := json.NewDecoder(resp.Body).Decode(&respCreds); err != nil {
|
||||
return ec2RoleCredRespBody{}, err
|
||||
}
|
||||
|
||||
if respCreds.Code != "Success" {
|
||||
// If an error code was returned something failed requesting the role.
|
||||
return ec2RoleCredRespBody{}, errors.New(respCreds.Message)
|
||||
}
|
||||
|
||||
return respCreds, nil
|
||||
}
|
||||
|
||||
// isLoopback identifies if a uri's host is on a loopback address
|
||||
func isLoopback(uri string) (bool, error) {
|
||||
u, err := url.Parse(uri)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
|
||||
host := u.Hostname()
|
||||
if len(host) == 0 {
|
||||
return false, fmt.Errorf("can't parse host from uri: %s", uri)
|
||||
}
|
||||
|
||||
ips, err := net.LookupHost(host)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
for _, ip := range ips {
|
||||
if !net.ParseIP(ip).IsLoopback() {
|
||||
return false, nil
|
||||
}
|
||||
}
|
||||
|
||||
return true, nil
|
||||
}
|
||||
Generated
Vendored
+77
@@ -0,0 +1,77 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2017 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package credentials
|
||||
|
||||
import "strings"
|
||||
|
||||
// SignatureType is type of Authorization requested for a given HTTP request.
|
||||
type SignatureType int
|
||||
|
||||
// Different types of supported signatures - default is SignatureV4 or SignatureDefault.
|
||||
const (
|
||||
// SignatureDefault is always set to v4.
|
||||
SignatureDefault SignatureType = iota
|
||||
SignatureV4
|
||||
SignatureV2
|
||||
SignatureV4Streaming
|
||||
SignatureAnonymous // Anonymous signature signifies, no signature.
|
||||
)
|
||||
|
||||
// IsV2 - is signature SignatureV2?
|
||||
func (s SignatureType) IsV2() bool {
|
||||
return s == SignatureV2
|
||||
}
|
||||
|
||||
// IsV4 - is signature SignatureV4?
|
||||
func (s SignatureType) IsV4() bool {
|
||||
return s == SignatureV4 || s == SignatureDefault
|
||||
}
|
||||
|
||||
// IsStreamingV4 - is signature SignatureV4Streaming?
|
||||
func (s SignatureType) IsStreamingV4() bool {
|
||||
return s == SignatureV4Streaming
|
||||
}
|
||||
|
||||
// IsAnonymous - is signature empty?
|
||||
func (s SignatureType) IsAnonymous() bool {
|
||||
return s == SignatureAnonymous
|
||||
}
|
||||
|
||||
// Stringer humanized version of signature type,
|
||||
// strings returned here are case insensitive.
|
||||
func (s SignatureType) String() string {
|
||||
if s.IsV2() {
|
||||
return "S3v2"
|
||||
} else if s.IsV4() {
|
||||
return "S3v4"
|
||||
} else if s.IsStreamingV4() {
|
||||
return "S3v4Streaming"
|
||||
}
|
||||
return "Anonymous"
|
||||
}
|
||||
|
||||
func parseSignatureType(str string) SignatureType {
|
||||
if strings.EqualFold(str, "S3v4") {
|
||||
return SignatureV4
|
||||
} else if strings.EqualFold(str, "S3v2") {
|
||||
return SignatureV2
|
||||
} else if strings.EqualFold(str, "S3v4Streaming") {
|
||||
return SignatureV4Streaming
|
||||
}
|
||||
return SignatureAnonymous
|
||||
}
|
||||
+72
@@ -0,0 +1,72 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2017 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package credentials
|
||||
|
||||
// A Static is a set of credentials which are set programmatically,
|
||||
// and will never expire.
|
||||
type Static struct {
|
||||
Value
|
||||
}
|
||||
|
||||
// NewStaticV2 returns a pointer to a new Credentials object
|
||||
// wrapping a static credentials value provider, signature is
|
||||
// set to v2. If access and secret are not specified then
|
||||
// regardless of signature type set it Value will return
|
||||
// as anonymous.
|
||||
func NewStaticV2(id, secret, token string) *Credentials {
|
||||
return NewStatic(id, secret, token, SignatureV2)
|
||||
}
|
||||
|
||||
// NewStaticV4 is similar to NewStaticV2 with similar considerations.
|
||||
func NewStaticV4(id, secret, token string) *Credentials {
|
||||
return NewStatic(id, secret, token, SignatureV4)
|
||||
}
|
||||
|
||||
// NewStatic returns a pointer to a new Credentials object
|
||||
// wrapping a static credentials value provider.
|
||||
func NewStatic(id, secret, token string, signerType SignatureType) *Credentials {
|
||||
return New(&Static{
|
||||
Value: Value{
|
||||
AccessKeyID: id,
|
||||
SecretAccessKey: secret,
|
||||
SessionToken: token,
|
||||
SignerType: signerType,
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
// Retrieve returns the static credentials.
|
||||
func (s *Static) Retrieve() (Value, error) {
|
||||
if s.AccessKeyID == "" || s.SecretAccessKey == "" {
|
||||
// Anonymous is not an error
|
||||
return Value{SignerType: SignatureAnonymous}, nil
|
||||
}
|
||||
return s.Value, nil
|
||||
}
|
||||
|
||||
// RetrieveWithCredContext returns the static credentials.
|
||||
func (s *Static) RetrieveWithCredContext(_ *CredContext) (Value, error) {
|
||||
return s.Retrieve()
|
||||
}
|
||||
|
||||
// IsExpired returns if the credentials are expired.
|
||||
//
|
||||
// For Static, the credentials never expired.
|
||||
func (s *Static) IsExpired() bool {
|
||||
return false
|
||||
}
|
||||
Generated
Vendored
+203
@@ -0,0 +1,203 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2019-2022 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package credentials
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/xml"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// AssumedRoleUser - The identifiers for the temporary security credentials that
|
||||
// the operation returns. Please also see https://docs.aws.amazon.com/goto/WebAPI/sts-2011-06-15/AssumedRoleUser
|
||||
type AssumedRoleUser struct {
|
||||
Arn string
|
||||
AssumedRoleID string `xml:"AssumeRoleId"`
|
||||
}
|
||||
|
||||
// AssumeRoleWithClientGrantsResponse contains the result of successful AssumeRoleWithClientGrants request.
|
||||
type AssumeRoleWithClientGrantsResponse struct {
|
||||
XMLName xml.Name `xml:"https://sts.amazonaws.com/doc/2011-06-15/ AssumeRoleWithClientGrantsResponse" json:"-"`
|
||||
Result ClientGrantsResult `xml:"AssumeRoleWithClientGrantsResult"`
|
||||
ResponseMetadata struct {
|
||||
RequestID string `xml:"RequestId,omitempty"`
|
||||
} `xml:"ResponseMetadata,omitempty"`
|
||||
}
|
||||
|
||||
// ClientGrantsResult - Contains the response to a successful AssumeRoleWithClientGrants
|
||||
// request, including temporary credentials that can be used to make MinIO API requests.
|
||||
type ClientGrantsResult struct {
|
||||
AssumedRoleUser AssumedRoleUser `xml:",omitempty"`
|
||||
Audience string `xml:",omitempty"`
|
||||
Credentials struct {
|
||||
AccessKey string `xml:"AccessKeyId" json:"accessKey,omitempty"`
|
||||
SecretKey string `xml:"SecretAccessKey" json:"secretKey,omitempty"`
|
||||
Expiration time.Time `xml:"Expiration" json:"expiration,omitempty"`
|
||||
SessionToken string `xml:"SessionToken" json:"sessionToken,omitempty"`
|
||||
} `xml:",omitempty"`
|
||||
PackedPolicySize int `xml:",omitempty"`
|
||||
Provider string `xml:",omitempty"`
|
||||
SubjectFromClientGrantsToken string `xml:",omitempty"`
|
||||
}
|
||||
|
||||
// ClientGrantsToken - client grants token with expiry.
|
||||
type ClientGrantsToken struct {
|
||||
Token string
|
||||
Expiry int
|
||||
}
|
||||
|
||||
// A STSClientGrants retrieves credentials from MinIO service, and keeps track if
|
||||
// those credentials are expired.
|
||||
type STSClientGrants struct {
|
||||
Expiry
|
||||
|
||||
// Optional http Client to use when connecting to MinIO STS service.
|
||||
// (overrides default client in CredContext)
|
||||
Client *http.Client
|
||||
|
||||
// MinIO endpoint to fetch STS credentials.
|
||||
STSEndpoint string
|
||||
|
||||
// getClientGrantsTokenExpiry function to retrieve tokens
|
||||
// from IDP This function should return two values one is
|
||||
// accessToken which is a self contained access token (JWT)
|
||||
// and second return value is the expiry associated with
|
||||
// this token. This is a customer provided function and
|
||||
// is mandatory.
|
||||
GetClientGrantsTokenExpiry func() (*ClientGrantsToken, error)
|
||||
}
|
||||
|
||||
// NewSTSClientGrants returns a pointer to a new
|
||||
// Credentials object wrapping the STSClientGrants.
|
||||
func NewSTSClientGrants(stsEndpoint string, getClientGrantsTokenExpiry func() (*ClientGrantsToken, error)) (*Credentials, error) {
|
||||
if getClientGrantsTokenExpiry == nil {
|
||||
return nil, errors.New("Client grants access token and expiry retrieval function should be defined")
|
||||
}
|
||||
return New(&STSClientGrants{
|
||||
STSEndpoint: stsEndpoint,
|
||||
GetClientGrantsTokenExpiry: getClientGrantsTokenExpiry,
|
||||
}), nil
|
||||
}
|
||||
|
||||
func getClientGrantsCredentials(clnt *http.Client, endpoint string,
|
||||
getClientGrantsTokenExpiry func() (*ClientGrantsToken, error),
|
||||
) (AssumeRoleWithClientGrantsResponse, error) {
|
||||
accessToken, err := getClientGrantsTokenExpiry()
|
||||
if err != nil {
|
||||
return AssumeRoleWithClientGrantsResponse{}, err
|
||||
}
|
||||
|
||||
v := url.Values{}
|
||||
v.Set("Action", "AssumeRoleWithClientGrants")
|
||||
v.Set("Token", accessToken.Token)
|
||||
v.Set("DurationSeconds", fmt.Sprintf("%d", accessToken.Expiry))
|
||||
v.Set("Version", STSVersion)
|
||||
|
||||
u, err := url.Parse(endpoint)
|
||||
if err != nil {
|
||||
return AssumeRoleWithClientGrantsResponse{}, err
|
||||
}
|
||||
|
||||
req, err := http.NewRequest(http.MethodPost, u.String(), strings.NewReader(v.Encode()))
|
||||
if err != nil {
|
||||
return AssumeRoleWithClientGrantsResponse{}, err
|
||||
}
|
||||
|
||||
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||
|
||||
resp, err := clnt.Do(req)
|
||||
if err != nil {
|
||||
return AssumeRoleWithClientGrantsResponse{}, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
var errResp ErrorResponse
|
||||
buf, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
return AssumeRoleWithClientGrantsResponse{}, err
|
||||
}
|
||||
_, err = xmlDecodeAndBody(bytes.NewReader(buf), &errResp)
|
||||
if err != nil {
|
||||
var s3Err Error
|
||||
if _, err = xmlDecodeAndBody(bytes.NewReader(buf), &s3Err); err != nil {
|
||||
return AssumeRoleWithClientGrantsResponse{}, err
|
||||
}
|
||||
errResp.RequestID = s3Err.RequestID
|
||||
errResp.STSError.Code = s3Err.Code
|
||||
errResp.STSError.Message = s3Err.Message
|
||||
}
|
||||
return AssumeRoleWithClientGrantsResponse{}, errResp
|
||||
}
|
||||
|
||||
a := AssumeRoleWithClientGrantsResponse{}
|
||||
if err = xml.NewDecoder(resp.Body).Decode(&a); err != nil {
|
||||
return AssumeRoleWithClientGrantsResponse{}, err
|
||||
}
|
||||
return a, nil
|
||||
}
|
||||
|
||||
// RetrieveWithCredContext is like Retrieve() with cred context
|
||||
func (m *STSClientGrants) RetrieveWithCredContext(cc *CredContext) (Value, error) {
|
||||
if cc == nil {
|
||||
cc = defaultCredContext
|
||||
}
|
||||
|
||||
client := m.Client
|
||||
if client == nil {
|
||||
client = cc.Client
|
||||
}
|
||||
if client == nil {
|
||||
client = defaultCredContext.Client
|
||||
}
|
||||
|
||||
stsEndpoint := m.STSEndpoint
|
||||
if stsEndpoint == "" {
|
||||
stsEndpoint = cc.Endpoint
|
||||
}
|
||||
if stsEndpoint == "" {
|
||||
return Value{}, errors.New("STS endpoint unknown")
|
||||
}
|
||||
|
||||
a, err := getClientGrantsCredentials(client, stsEndpoint, m.GetClientGrantsTokenExpiry)
|
||||
if err != nil {
|
||||
return Value{}, err
|
||||
}
|
||||
|
||||
// Expiry window is set to 10secs.
|
||||
m.SetExpiration(a.Result.Credentials.Expiration, DefaultExpiryWindow)
|
||||
|
||||
return Value{
|
||||
AccessKeyID: a.Result.Credentials.AccessKey,
|
||||
SecretAccessKey: a.Result.Credentials.SecretKey,
|
||||
SessionToken: a.Result.Credentials.SessionToken,
|
||||
Expiration: a.Result.Credentials.Expiration,
|
||||
SignerType: SignatureV4,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Retrieve retrieves credentials from the MinIO service.
|
||||
// Error will be returned if the request fails.
|
||||
func (m *STSClientGrants) Retrieve() (Value, error) {
|
||||
return m.RetrieveWithCredContext(nil)
|
||||
}
|
||||
Generated
Vendored
+173
@@ -0,0 +1,173 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2015-2022 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package credentials
|
||||
|
||||
import (
|
||||
"encoding/xml"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"time"
|
||||
)
|
||||
|
||||
// CustomTokenResult - Contains temporary creds and user metadata.
|
||||
type CustomTokenResult struct {
|
||||
Credentials struct {
|
||||
AccessKey string `xml:"AccessKeyId"`
|
||||
SecretKey string `xml:"SecretAccessKey"`
|
||||
Expiration time.Time `xml:"Expiration"`
|
||||
SessionToken string `xml:"SessionToken"`
|
||||
} `xml:",omitempty"`
|
||||
|
||||
AssumedUser string `xml:",omitempty"`
|
||||
}
|
||||
|
||||
// AssumeRoleWithCustomTokenResponse contains the result of a successful
|
||||
// AssumeRoleWithCustomToken request.
|
||||
type AssumeRoleWithCustomTokenResponse struct {
|
||||
XMLName xml.Name `xml:"https://sts.amazonaws.com/doc/2011-06-15/ AssumeRoleWithCustomTokenResponse" json:"-"`
|
||||
Result CustomTokenResult `xml:"AssumeRoleWithCustomTokenResult"`
|
||||
Metadata struct {
|
||||
RequestID string `xml:"RequestId,omitempty"`
|
||||
} `xml:"ResponseMetadata,omitempty"`
|
||||
}
|
||||
|
||||
// CustomTokenIdentity - satisfies the Provider interface, and retrieves
|
||||
// credentials from MinIO using the AssumeRoleWithCustomToken STS API.
|
||||
type CustomTokenIdentity struct {
|
||||
Expiry
|
||||
|
||||
// Optional http Client to use when connecting to MinIO STS service.
|
||||
// (overrides default client in CredContext)
|
||||
Client *http.Client
|
||||
|
||||
// MinIO server STS endpoint to fetch STS credentials.
|
||||
STSEndpoint string
|
||||
|
||||
// The custom token to use with the request.
|
||||
Token string
|
||||
|
||||
// RoleArn associated with the identity
|
||||
RoleArn string
|
||||
|
||||
// RequestedExpiry is to set the validity of the generated credentials
|
||||
// (this value bounded by server).
|
||||
RequestedExpiry time.Duration
|
||||
}
|
||||
|
||||
// RetrieveWithCredContext with Retrieve optionally cred context
|
||||
func (c *CustomTokenIdentity) RetrieveWithCredContext(cc *CredContext) (value Value, err error) {
|
||||
if cc == nil {
|
||||
cc = defaultCredContext
|
||||
}
|
||||
|
||||
stsEndpoint := c.STSEndpoint
|
||||
if stsEndpoint == "" {
|
||||
stsEndpoint = cc.Endpoint
|
||||
}
|
||||
if stsEndpoint == "" {
|
||||
return Value{}, errors.New("STS endpoint unknown")
|
||||
}
|
||||
|
||||
u, err := url.Parse(stsEndpoint)
|
||||
if err != nil {
|
||||
return value, err
|
||||
}
|
||||
|
||||
v := url.Values{}
|
||||
v.Set("Action", "AssumeRoleWithCustomToken")
|
||||
v.Set("Version", STSVersion)
|
||||
v.Set("RoleArn", c.RoleArn)
|
||||
v.Set("Token", c.Token)
|
||||
if c.RequestedExpiry != 0 {
|
||||
v.Set("DurationSeconds", fmt.Sprintf("%d", int(c.RequestedExpiry.Seconds())))
|
||||
}
|
||||
|
||||
u.RawQuery = v.Encode()
|
||||
|
||||
req, err := http.NewRequest(http.MethodPost, u.String(), nil)
|
||||
if err != nil {
|
||||
return value, err
|
||||
}
|
||||
|
||||
client := c.Client
|
||||
if client == nil {
|
||||
client = cc.Client
|
||||
}
|
||||
if client == nil {
|
||||
client = defaultCredContext.Client
|
||||
}
|
||||
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return value, err
|
||||
}
|
||||
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return value, errors.New(resp.Status)
|
||||
}
|
||||
|
||||
r := AssumeRoleWithCustomTokenResponse{}
|
||||
if err = xml.NewDecoder(resp.Body).Decode(&r); err != nil {
|
||||
return
|
||||
}
|
||||
|
||||
cr := r.Result.Credentials
|
||||
c.SetExpiration(cr.Expiration, DefaultExpiryWindow)
|
||||
return Value{
|
||||
AccessKeyID: cr.AccessKey,
|
||||
SecretAccessKey: cr.SecretKey,
|
||||
SessionToken: cr.SessionToken,
|
||||
Expiration: cr.Expiration,
|
||||
SignerType: SignatureV4,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Retrieve - to satisfy Provider interface; fetches credentials from MinIO.
|
||||
func (c *CustomTokenIdentity) Retrieve() (value Value, err error) {
|
||||
return c.RetrieveWithCredContext(nil)
|
||||
}
|
||||
|
||||
// NewCustomTokenCredentials - returns credentials using the
|
||||
// AssumeRoleWithCustomToken STS API.
|
||||
func NewCustomTokenCredentials(stsEndpoint, token, roleArn string, optFuncs ...CustomTokenOpt) (*Credentials, error) {
|
||||
c := CustomTokenIdentity{
|
||||
STSEndpoint: stsEndpoint,
|
||||
Token: token,
|
||||
RoleArn: roleArn,
|
||||
}
|
||||
for _, optFunc := range optFuncs {
|
||||
optFunc(&c)
|
||||
}
|
||||
return New(&c), nil
|
||||
}
|
||||
|
||||
// CustomTokenOpt is a function type to configure the custom-token based
|
||||
// credentials using NewCustomTokenCredentials.
|
||||
type CustomTokenOpt func(*CustomTokenIdentity)
|
||||
|
||||
// CustomTokenValidityOpt sets the validity duration of the requested
|
||||
// credentials. This value is ignored if the server enforces a lower validity
|
||||
// period.
|
||||
func CustomTokenValidityOpt(d time.Duration) CustomTokenOpt {
|
||||
return func(c *CustomTokenIdentity) {
|
||||
c.RequestedExpiry = d
|
||||
}
|
||||
}
|
||||
Generated
Vendored
+216
@@ -0,0 +1,216 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2019-2022 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package credentials
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/xml"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// AssumeRoleWithLDAPResponse contains the result of successful
|
||||
// AssumeRoleWithLDAPIdentity request
|
||||
type AssumeRoleWithLDAPResponse struct {
|
||||
XMLName xml.Name `xml:"https://sts.amazonaws.com/doc/2011-06-15/ AssumeRoleWithLDAPIdentityResponse" json:"-"`
|
||||
Result LDAPIdentityResult `xml:"AssumeRoleWithLDAPIdentityResult"`
|
||||
ResponseMetadata struct {
|
||||
RequestID string `xml:"RequestId,omitempty"`
|
||||
} `xml:"ResponseMetadata,omitempty"`
|
||||
}
|
||||
|
||||
// LDAPIdentityResult - contains credentials for a successful
|
||||
// AssumeRoleWithLDAPIdentity request.
|
||||
type LDAPIdentityResult struct {
|
||||
Credentials struct {
|
||||
AccessKey string `xml:"AccessKeyId" json:"accessKey,omitempty"`
|
||||
SecretKey string `xml:"SecretAccessKey" json:"secretKey,omitempty"`
|
||||
Expiration time.Time `xml:"Expiration" json:"expiration,omitempty"`
|
||||
SessionToken string `xml:"SessionToken" json:"sessionToken,omitempty"`
|
||||
} `xml:",omitempty"`
|
||||
|
||||
SubjectFromToken string `xml:",omitempty"`
|
||||
}
|
||||
|
||||
// LDAPIdentity retrieves credentials from MinIO
|
||||
type LDAPIdentity struct {
|
||||
Expiry
|
||||
|
||||
// Optional http Client to use when connecting to MinIO STS service.
|
||||
// (overrides default client in CredContext)
|
||||
Client *http.Client
|
||||
|
||||
// Exported STS endpoint to fetch STS credentials.
|
||||
STSEndpoint string
|
||||
|
||||
// LDAP username/password used to fetch LDAP STS credentials.
|
||||
LDAPUsername, LDAPPassword string
|
||||
|
||||
// Session policy to apply to the generated credentials. Leave empty to
|
||||
// use the full access policy available to the user.
|
||||
Policy string
|
||||
|
||||
// RequestedExpiry is the configured expiry duration for credentials
|
||||
// requested from LDAP.
|
||||
RequestedExpiry time.Duration
|
||||
}
|
||||
|
||||
// NewLDAPIdentity returns new credentials object that uses LDAP
|
||||
// Identity.
|
||||
func NewLDAPIdentity(stsEndpoint, ldapUsername, ldapPassword string, optFuncs ...LDAPIdentityOpt) (*Credentials, error) {
|
||||
l := LDAPIdentity{
|
||||
STSEndpoint: stsEndpoint,
|
||||
LDAPUsername: ldapUsername,
|
||||
LDAPPassword: ldapPassword,
|
||||
}
|
||||
for _, optFunc := range optFuncs {
|
||||
optFunc(&l)
|
||||
}
|
||||
return New(&l), nil
|
||||
}
|
||||
|
||||
// LDAPIdentityOpt is a function type used to configured the LDAPIdentity
|
||||
// instance.
|
||||
type LDAPIdentityOpt func(*LDAPIdentity)
|
||||
|
||||
// LDAPIdentityPolicyOpt sets the session policy for requested credentials.
|
||||
func LDAPIdentityPolicyOpt(policy string) LDAPIdentityOpt {
|
||||
return func(k *LDAPIdentity) {
|
||||
k.Policy = policy
|
||||
}
|
||||
}
|
||||
|
||||
// LDAPIdentityExpiryOpt sets the expiry duration for requested credentials.
|
||||
func LDAPIdentityExpiryOpt(d time.Duration) LDAPIdentityOpt {
|
||||
return func(k *LDAPIdentity) {
|
||||
k.RequestedExpiry = d
|
||||
}
|
||||
}
|
||||
|
||||
// NewLDAPIdentityWithSessionPolicy returns new credentials object that uses
|
||||
// LDAP Identity with a specified session policy. The `policy` parameter must be
|
||||
// a JSON string specifying the policy document.
|
||||
//
|
||||
// Deprecated: Use the `LDAPIdentityPolicyOpt` with `NewLDAPIdentity` instead.
|
||||
func NewLDAPIdentityWithSessionPolicy(stsEndpoint, ldapUsername, ldapPassword, policy string) (*Credentials, error) {
|
||||
return New(&LDAPIdentity{
|
||||
STSEndpoint: stsEndpoint,
|
||||
LDAPUsername: ldapUsername,
|
||||
LDAPPassword: ldapPassword,
|
||||
Policy: policy,
|
||||
}), nil
|
||||
}
|
||||
|
||||
// RetrieveWithCredContext gets the credential by calling the MinIO STS API for
|
||||
// LDAP on the configured stsEndpoint.
|
||||
func (k *LDAPIdentity) RetrieveWithCredContext(cc *CredContext) (value Value, err error) {
|
||||
if cc == nil {
|
||||
cc = defaultCredContext
|
||||
}
|
||||
|
||||
stsEndpoint := k.STSEndpoint
|
||||
if stsEndpoint == "" {
|
||||
stsEndpoint = cc.Endpoint
|
||||
}
|
||||
if stsEndpoint == "" {
|
||||
return Value{}, errors.New("STS endpoint unknown")
|
||||
}
|
||||
|
||||
u, err := url.Parse(stsEndpoint)
|
||||
if err != nil {
|
||||
return value, err
|
||||
}
|
||||
|
||||
v := url.Values{}
|
||||
v.Set("Action", "AssumeRoleWithLDAPIdentity")
|
||||
v.Set("Version", STSVersion)
|
||||
v.Set("LDAPUsername", k.LDAPUsername)
|
||||
v.Set("LDAPPassword", k.LDAPPassword)
|
||||
if k.Policy != "" {
|
||||
v.Set("Policy", k.Policy)
|
||||
}
|
||||
if k.RequestedExpiry != 0 {
|
||||
v.Set("DurationSeconds", fmt.Sprintf("%d", int(k.RequestedExpiry.Seconds())))
|
||||
}
|
||||
|
||||
req, err := http.NewRequest(http.MethodPost, u.String(), strings.NewReader(v.Encode()))
|
||||
if err != nil {
|
||||
return value, err
|
||||
}
|
||||
|
||||
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||
|
||||
client := k.Client
|
||||
if client == nil {
|
||||
client = cc.Client
|
||||
}
|
||||
if client == nil {
|
||||
client = defaultCredContext.Client
|
||||
}
|
||||
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return value, err
|
||||
}
|
||||
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
var errResp ErrorResponse
|
||||
buf, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
return value, err
|
||||
}
|
||||
_, err = xmlDecodeAndBody(bytes.NewReader(buf), &errResp)
|
||||
if err != nil {
|
||||
var s3Err Error
|
||||
if _, err = xmlDecodeAndBody(bytes.NewReader(buf), &s3Err); err != nil {
|
||||
return value, err
|
||||
}
|
||||
errResp.RequestID = s3Err.RequestID
|
||||
errResp.STSError.Code = s3Err.Code
|
||||
errResp.STSError.Message = s3Err.Message
|
||||
}
|
||||
return value, errResp
|
||||
}
|
||||
|
||||
r := AssumeRoleWithLDAPResponse{}
|
||||
if err = xml.NewDecoder(resp.Body).Decode(&r); err != nil {
|
||||
return
|
||||
}
|
||||
|
||||
cr := r.Result.Credentials
|
||||
k.SetExpiration(cr.Expiration, DefaultExpiryWindow)
|
||||
return Value{
|
||||
AccessKeyID: cr.AccessKey,
|
||||
SecretAccessKey: cr.SecretKey,
|
||||
SessionToken: cr.SessionToken,
|
||||
Expiration: cr.Expiration,
|
||||
SignerType: SignatureV4,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Retrieve gets the credential by calling the MinIO STS API for
|
||||
// LDAP on the configured stsEndpoint.
|
||||
func (k *LDAPIdentity) Retrieve() (value Value, err error) {
|
||||
return k.RetrieveWithCredContext(defaultCredContext)
|
||||
}
|
||||
Generated
Vendored
+226
@@ -0,0 +1,226 @@
|
||||
// MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
// Copyright 2021 MinIO, Inc.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
package credentials
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/tls"
|
||||
"encoding/xml"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strconv"
|
||||
"time"
|
||||
)
|
||||
|
||||
// CertificateIdentityOption is an optional AssumeRoleWithCertificate
|
||||
// parameter - e.g. a custom HTTP transport configuration or S3 credental
|
||||
// livetime.
|
||||
type CertificateIdentityOption func(*STSCertificateIdentity)
|
||||
|
||||
// CertificateIdentityWithTransport returns a CertificateIdentityOption that
|
||||
// customizes the STSCertificateIdentity with the given http.RoundTripper.
|
||||
func CertificateIdentityWithTransport(t http.RoundTripper) CertificateIdentityOption {
|
||||
return CertificateIdentityOption(func(i *STSCertificateIdentity) {
|
||||
if i.Client == nil {
|
||||
i.Client = &http.Client{}
|
||||
}
|
||||
i.Client.Transport = t
|
||||
})
|
||||
}
|
||||
|
||||
// CertificateIdentityWithExpiry returns a CertificateIdentityOption that
|
||||
// customizes the STSCertificateIdentity with the given livetime.
|
||||
//
|
||||
// Fetched S3 credentials will have the given livetime if the STS server
|
||||
// allows such credentials.
|
||||
func CertificateIdentityWithExpiry(livetime time.Duration) CertificateIdentityOption {
|
||||
return CertificateIdentityOption(func(i *STSCertificateIdentity) { i.S3CredentialLivetime = livetime })
|
||||
}
|
||||
|
||||
// A STSCertificateIdentity retrieves S3 credentials from the MinIO STS API and
|
||||
// rotates those credentials once they expire.
|
||||
type STSCertificateIdentity struct {
|
||||
Expiry
|
||||
|
||||
// Optional http Client to use when connecting to MinIO STS service.
|
||||
// (overrides default client in CredContext)
|
||||
Client *http.Client
|
||||
|
||||
// STSEndpoint is the base URL endpoint of the STS API.
|
||||
// For example, https://minio.local:9000
|
||||
STSEndpoint string
|
||||
|
||||
// S3CredentialLivetime is the duration temp. S3 access
|
||||
// credentials should be valid.
|
||||
//
|
||||
// It represents the access credential livetime requested
|
||||
// by the client. The STS server may choose to issue
|
||||
// temp. S3 credentials that have a different - usually
|
||||
// shorter - livetime.
|
||||
//
|
||||
// The default livetime is one hour.
|
||||
S3CredentialLivetime time.Duration
|
||||
|
||||
// Certificate is the client certificate that is used for
|
||||
// STS authentication.
|
||||
Certificate tls.Certificate
|
||||
}
|
||||
|
||||
// NewSTSCertificateIdentity returns a STSCertificateIdentity that authenticates
|
||||
// to the given STS endpoint with the given TLS certificate and retrieves and
|
||||
// rotates S3 credentials.
|
||||
func NewSTSCertificateIdentity(endpoint string, certificate tls.Certificate, options ...CertificateIdentityOption) (*Credentials, error) {
|
||||
identity := &STSCertificateIdentity{
|
||||
STSEndpoint: endpoint,
|
||||
Certificate: certificate,
|
||||
}
|
||||
for _, option := range options {
|
||||
option(identity)
|
||||
}
|
||||
return New(identity), nil
|
||||
}
|
||||
|
||||
// RetrieveWithCredContext is Retrieve with cred context
|
||||
func (i *STSCertificateIdentity) RetrieveWithCredContext(cc *CredContext) (Value, error) {
|
||||
if cc == nil {
|
||||
cc = defaultCredContext
|
||||
}
|
||||
|
||||
stsEndpoint := i.STSEndpoint
|
||||
if stsEndpoint == "" {
|
||||
stsEndpoint = cc.Endpoint
|
||||
}
|
||||
if stsEndpoint == "" {
|
||||
return Value{}, errors.New("STS endpoint unknown")
|
||||
}
|
||||
|
||||
endpointURL, err := url.Parse(stsEndpoint)
|
||||
if err != nil {
|
||||
return Value{}, err
|
||||
}
|
||||
livetime := i.S3CredentialLivetime
|
||||
if livetime == 0 {
|
||||
livetime = 1 * time.Hour
|
||||
}
|
||||
|
||||
queryValues := url.Values{}
|
||||
queryValues.Set("Action", "AssumeRoleWithCertificate")
|
||||
queryValues.Set("Version", STSVersion)
|
||||
endpointURL.RawQuery = queryValues.Encode()
|
||||
|
||||
req, err := http.NewRequest(http.MethodPost, endpointURL.String(), nil)
|
||||
if err != nil {
|
||||
return Value{}, err
|
||||
}
|
||||
if req.Form == nil {
|
||||
req.Form = url.Values{}
|
||||
}
|
||||
req.Form.Add("DurationSeconds", strconv.FormatUint(uint64(livetime.Seconds()), 10))
|
||||
|
||||
client := i.Client
|
||||
if client == nil {
|
||||
client = cc.Client
|
||||
}
|
||||
if client == nil {
|
||||
client = defaultCredContext.Client
|
||||
}
|
||||
|
||||
tr, ok := client.Transport.(*http.Transport)
|
||||
if !ok {
|
||||
return Value{}, fmt.Errorf("CredContext should contain an http.Transport value")
|
||||
}
|
||||
|
||||
// Clone the HTTP transport (patch the TLS client certificate)
|
||||
trCopy := tr.Clone()
|
||||
trCopy.TLSClientConfig.Certificates = []tls.Certificate{i.Certificate}
|
||||
|
||||
// Clone the HTTP client (patch the HTTP transport)
|
||||
clientCopy := *client
|
||||
clientCopy.Transport = trCopy
|
||||
|
||||
resp, err := clientCopy.Do(req)
|
||||
if err != nil {
|
||||
return Value{}, err
|
||||
}
|
||||
if resp.Body != nil {
|
||||
defer resp.Body.Close()
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
var errResp ErrorResponse
|
||||
buf, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
return Value{}, err
|
||||
}
|
||||
_, err = xmlDecodeAndBody(bytes.NewReader(buf), &errResp)
|
||||
if err != nil {
|
||||
var s3Err Error
|
||||
if _, err = xmlDecodeAndBody(bytes.NewReader(buf), &s3Err); err != nil {
|
||||
return Value{}, err
|
||||
}
|
||||
errResp.RequestID = s3Err.RequestID
|
||||
errResp.STSError.Code = s3Err.Code
|
||||
errResp.STSError.Message = s3Err.Message
|
||||
}
|
||||
return Value{}, errResp
|
||||
}
|
||||
|
||||
const MaxSize = 10 * 1 << 20
|
||||
var body io.Reader = resp.Body
|
||||
if resp.ContentLength > 0 && resp.ContentLength < MaxSize {
|
||||
body = io.LimitReader(body, resp.ContentLength)
|
||||
} else {
|
||||
body = io.LimitReader(body, MaxSize)
|
||||
}
|
||||
|
||||
var response assumeRoleWithCertificateResponse
|
||||
if err = xml.NewDecoder(body).Decode(&response); err != nil {
|
||||
return Value{}, err
|
||||
}
|
||||
i.SetExpiration(response.Result.Credentials.Expiration, DefaultExpiryWindow)
|
||||
return Value{
|
||||
AccessKeyID: response.Result.Credentials.AccessKey,
|
||||
SecretAccessKey: response.Result.Credentials.SecretKey,
|
||||
SessionToken: response.Result.Credentials.SessionToken,
|
||||
Expiration: response.Result.Credentials.Expiration,
|
||||
SignerType: SignatureDefault,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Retrieve fetches a new set of S3 credentials from the configured STS API endpoint.
|
||||
func (i *STSCertificateIdentity) Retrieve() (Value, error) {
|
||||
return i.RetrieveWithCredContext(defaultCredContext)
|
||||
}
|
||||
|
||||
// Expiration returns the expiration time of the current S3 credentials.
|
||||
func (i *STSCertificateIdentity) Expiration() time.Time { return i.expiration }
|
||||
|
||||
type assumeRoleWithCertificateResponse struct {
|
||||
XMLName xml.Name `xml:"https://sts.amazonaws.com/doc/2011-06-15/ AssumeRoleWithCertificateResponse" json:"-"`
|
||||
Result struct {
|
||||
Credentials struct {
|
||||
AccessKey string `xml:"AccessKeyId" json:"accessKey,omitempty"`
|
||||
SecretKey string `xml:"SecretAccessKey" json:"secretKey,omitempty"`
|
||||
Expiration time.Time `xml:"Expiration" json:"expiration,omitempty"`
|
||||
SessionToken string `xml:"SessionToken" json:"sessionToken,omitempty"`
|
||||
} `xml:"Credentials" json:"credentials,omitempty"`
|
||||
} `xml:"AssumeRoleWithCertificateResult"`
|
||||
ResponseMetadata struct {
|
||||
RequestID string `xml:"RequestId,omitempty"`
|
||||
} `xml:"ResponseMetadata,omitempty"`
|
||||
}
|
||||
Generated
Vendored
+265
@@ -0,0 +1,265 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2019-2022 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package credentials
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/xml"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// AssumeRoleWithWebIdentityResponse contains the result of successful AssumeRoleWithWebIdentity request.
|
||||
type AssumeRoleWithWebIdentityResponse struct {
|
||||
XMLName xml.Name `xml:"https://sts.amazonaws.com/doc/2011-06-15/ AssumeRoleWithWebIdentityResponse" json:"-"`
|
||||
Result WebIdentityResult `xml:"AssumeRoleWithWebIdentityResult"`
|
||||
ResponseMetadata struct {
|
||||
RequestID string `xml:"RequestId,omitempty"`
|
||||
} `xml:"ResponseMetadata,omitempty"`
|
||||
}
|
||||
|
||||
// WebIdentityResult - Contains the response to a successful AssumeRoleWithWebIdentity
|
||||
// request, including temporary credentials that can be used to make MinIO API requests.
|
||||
type WebIdentityResult struct {
|
||||
AssumedRoleUser AssumedRoleUser `xml:",omitempty"`
|
||||
Audience string `xml:",omitempty"`
|
||||
Credentials struct {
|
||||
AccessKey string `xml:"AccessKeyId" json:"accessKey,omitempty"`
|
||||
SecretKey string `xml:"SecretAccessKey" json:"secretKey,omitempty"`
|
||||
Expiration time.Time `xml:"Expiration" json:"expiration,omitempty"`
|
||||
SessionToken string `xml:"SessionToken" json:"sessionToken,omitempty"`
|
||||
} `xml:",omitempty"`
|
||||
PackedPolicySize int `xml:",omitempty"`
|
||||
Provider string `xml:",omitempty"`
|
||||
SubjectFromWebIdentityToken string `xml:",omitempty"`
|
||||
}
|
||||
|
||||
// WebIdentityToken - web identity token with expiry.
|
||||
type WebIdentityToken struct {
|
||||
Token string
|
||||
AccessToken string
|
||||
RefreshToken string
|
||||
Expiry int
|
||||
}
|
||||
|
||||
// A STSWebIdentity retrieves credentials from MinIO service, and keeps track if
|
||||
// those credentials are expired.
|
||||
type STSWebIdentity struct {
|
||||
Expiry
|
||||
|
||||
// Optional http Client to use when connecting to MinIO STS service.
|
||||
// (overrides default client in CredContext)
|
||||
Client *http.Client
|
||||
|
||||
// Exported STS endpoint to fetch STS credentials.
|
||||
STSEndpoint string
|
||||
|
||||
// Exported GetWebIDTokenExpiry function which returns ID
|
||||
// tokens from IDP. This function should return two values
|
||||
// one is ID token which is a self contained ID token (JWT)
|
||||
// and second return value is the expiry associated with
|
||||
// this token.
|
||||
// This is a customer provided function and is mandatory.
|
||||
GetWebIDTokenExpiry func() (*WebIdentityToken, error)
|
||||
|
||||
// RoleARN is the Amazon Resource Name (ARN) of the role that the caller is
|
||||
// assuming.
|
||||
RoleARN string
|
||||
|
||||
// Policy is the policy where the credentials should be limited too.
|
||||
Policy string
|
||||
|
||||
// roleSessionName is the identifier for the assumed role session.
|
||||
roleSessionName string
|
||||
}
|
||||
|
||||
// NewSTSWebIdentity returns a pointer to a new
|
||||
// Credentials object wrapping the STSWebIdentity.
|
||||
func NewSTSWebIdentity(stsEndpoint string, getWebIDTokenExpiry func() (*WebIdentityToken, error), opts ...func(*STSWebIdentity)) (*Credentials, error) {
|
||||
if getWebIDTokenExpiry == nil {
|
||||
return nil, errors.New("Web ID token and expiry retrieval function should be defined")
|
||||
}
|
||||
i := &STSWebIdentity{
|
||||
STSEndpoint: stsEndpoint,
|
||||
GetWebIDTokenExpiry: getWebIDTokenExpiry,
|
||||
}
|
||||
for _, o := range opts {
|
||||
o(i)
|
||||
}
|
||||
return New(i), nil
|
||||
}
|
||||
|
||||
// NewKubernetesIdentity returns a pointer to a new
|
||||
// Credentials object using the Kubernetes service account
|
||||
func NewKubernetesIdentity(stsEndpoint string, opts ...func(*STSWebIdentity)) (*Credentials, error) {
|
||||
return NewSTSWebIdentity(stsEndpoint, func() (*WebIdentityToken, error) {
|
||||
token, err := os.ReadFile("/var/run/secrets/kubernetes.io/serviceaccount/token")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &WebIdentityToken{
|
||||
Token: string(token),
|
||||
}, nil
|
||||
}, opts...)
|
||||
}
|
||||
|
||||
// WithPolicy option will enforce that the returned credentials
|
||||
// will be scoped down to the specified policy
|
||||
func WithPolicy(policy string) func(*STSWebIdentity) {
|
||||
return func(i *STSWebIdentity) {
|
||||
i.Policy = policy
|
||||
}
|
||||
}
|
||||
|
||||
func getWebIdentityCredentials(clnt *http.Client, endpoint, roleARN, roleSessionName string, policy string,
|
||||
getWebIDTokenExpiry func() (*WebIdentityToken, error),
|
||||
) (AssumeRoleWithWebIdentityResponse, error) {
|
||||
idToken, err := getWebIDTokenExpiry()
|
||||
if err != nil {
|
||||
return AssumeRoleWithWebIdentityResponse{}, err
|
||||
}
|
||||
|
||||
v := url.Values{}
|
||||
v.Set("Action", "AssumeRoleWithWebIdentity")
|
||||
if len(roleARN) > 0 {
|
||||
v.Set("RoleArn", roleARN)
|
||||
|
||||
if len(roleSessionName) == 0 {
|
||||
roleSessionName = strconv.FormatInt(time.Now().UnixNano(), 10)
|
||||
}
|
||||
v.Set("RoleSessionName", roleSessionName)
|
||||
}
|
||||
v.Set("WebIdentityToken", idToken.Token)
|
||||
if idToken.AccessToken != "" {
|
||||
// Usually set when server is using extended userInfo endpoint.
|
||||
v.Set("WebIdentityAccessToken", idToken.AccessToken)
|
||||
}
|
||||
if idToken.RefreshToken != "" {
|
||||
// Usually set when server is using extended userInfo endpoint.
|
||||
v.Set("WebIdentityRefreshToken", idToken.RefreshToken)
|
||||
}
|
||||
if idToken.Expiry > 0 {
|
||||
v.Set("DurationSeconds", fmt.Sprintf("%d", idToken.Expiry))
|
||||
}
|
||||
if policy != "" {
|
||||
v.Set("Policy", policy)
|
||||
}
|
||||
v.Set("Version", STSVersion)
|
||||
|
||||
u, err := url.Parse(endpoint)
|
||||
if err != nil {
|
||||
return AssumeRoleWithWebIdentityResponse{}, err
|
||||
}
|
||||
|
||||
req, err := http.NewRequest(http.MethodPost, u.String(), strings.NewReader(v.Encode()))
|
||||
if err != nil {
|
||||
return AssumeRoleWithWebIdentityResponse{}, err
|
||||
}
|
||||
|
||||
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||
|
||||
resp, err := clnt.Do(req)
|
||||
if err != nil {
|
||||
return AssumeRoleWithWebIdentityResponse{}, err
|
||||
}
|
||||
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
var errResp ErrorResponse
|
||||
buf, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
return AssumeRoleWithWebIdentityResponse{}, err
|
||||
}
|
||||
_, err = xmlDecodeAndBody(bytes.NewReader(buf), &errResp)
|
||||
if err != nil {
|
||||
var s3Err Error
|
||||
if _, err = xmlDecodeAndBody(bytes.NewReader(buf), &s3Err); err != nil {
|
||||
return AssumeRoleWithWebIdentityResponse{}, err
|
||||
}
|
||||
errResp.RequestID = s3Err.RequestID
|
||||
errResp.STSError.Code = s3Err.Code
|
||||
errResp.STSError.Message = s3Err.Message
|
||||
}
|
||||
return AssumeRoleWithWebIdentityResponse{}, errResp
|
||||
}
|
||||
|
||||
a := AssumeRoleWithWebIdentityResponse{}
|
||||
if err = xml.NewDecoder(resp.Body).Decode(&a); err != nil {
|
||||
return AssumeRoleWithWebIdentityResponse{}, err
|
||||
}
|
||||
|
||||
return a, nil
|
||||
}
|
||||
|
||||
// RetrieveWithCredContext is like Retrieve with optional cred context.
|
||||
func (m *STSWebIdentity) RetrieveWithCredContext(cc *CredContext) (Value, error) {
|
||||
if cc == nil {
|
||||
cc = defaultCredContext
|
||||
}
|
||||
|
||||
client := m.Client
|
||||
if client == nil {
|
||||
client = cc.Client
|
||||
}
|
||||
if client == nil {
|
||||
client = defaultCredContext.Client
|
||||
}
|
||||
|
||||
stsEndpoint := m.STSEndpoint
|
||||
if stsEndpoint == "" {
|
||||
stsEndpoint = cc.Endpoint
|
||||
}
|
||||
if stsEndpoint == "" {
|
||||
return Value{}, errors.New("STS endpoint unknown")
|
||||
}
|
||||
|
||||
a, err := getWebIdentityCredentials(client, stsEndpoint, m.RoleARN, m.roleSessionName, m.Policy, m.GetWebIDTokenExpiry)
|
||||
if err != nil {
|
||||
return Value{}, err
|
||||
}
|
||||
|
||||
// Expiry window is set to 10secs.
|
||||
m.SetExpiration(a.Result.Credentials.Expiration, DefaultExpiryWindow)
|
||||
|
||||
return Value{
|
||||
AccessKeyID: a.Result.Credentials.AccessKey,
|
||||
SecretAccessKey: a.Result.Credentials.SecretKey,
|
||||
SessionToken: a.Result.Credentials.SessionToken,
|
||||
Expiration: a.Result.Credentials.Expiration,
|
||||
SignerType: SignatureV4,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Retrieve retrieves credentials from the MinIO service.
|
||||
// Error will be returned if the request fails.
|
||||
func (m *STSWebIdentity) Retrieve() (Value, error) {
|
||||
return m.RetrieveWithCredContext(nil)
|
||||
}
|
||||
|
||||
// Expiration returns the expiration time of the credentials
|
||||
func (m *STSWebIdentity) Expiration() time.Time {
|
||||
return m.expiration
|
||||
}
|
||||
+24
@@ -0,0 +1,24 @@
|
||||
//go:build !fips
|
||||
// +build !fips
|
||||
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2022 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package encrypt
|
||||
|
||||
// FIPS is true if 'fips' build tag was specified.
|
||||
const FIPS = false
|
||||
+24
@@ -0,0 +1,24 @@
|
||||
//go:build fips
|
||||
// +build fips
|
||||
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2022 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package encrypt
|
||||
|
||||
// FIPS is true if 'fips' build tag was specified.
|
||||
const FIPS = true
|
||||
+197
@@ -0,0 +1,197 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2018 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package encrypt
|
||||
|
||||
import (
|
||||
"crypto/md5"
|
||||
"encoding/base64"
|
||||
"errors"
|
||||
"net/http"
|
||||
|
||||
"github.com/goccy/go-json"
|
||||
"golang.org/x/crypto/argon2"
|
||||
)
|
||||
|
||||
const (
|
||||
// SseGenericHeader is the AWS SSE header used for SSE-S3 and SSE-KMS.
|
||||
SseGenericHeader = "X-Amz-Server-Side-Encryption"
|
||||
|
||||
// SseKmsKeyID is the AWS SSE-KMS key id.
|
||||
SseKmsKeyID = SseGenericHeader + "-Aws-Kms-Key-Id"
|
||||
// SseEncryptionContext is the AWS SSE-KMS Encryption Context data.
|
||||
SseEncryptionContext = SseGenericHeader + "-Context"
|
||||
|
||||
// SseCustomerAlgorithm is the AWS SSE-C algorithm HTTP header key.
|
||||
SseCustomerAlgorithm = SseGenericHeader + "-Customer-Algorithm"
|
||||
// SseCustomerKey is the AWS SSE-C encryption key HTTP header key.
|
||||
SseCustomerKey = SseGenericHeader + "-Customer-Key"
|
||||
// SseCustomerKeyMD5 is the AWS SSE-C encryption key MD5 HTTP header key.
|
||||
SseCustomerKeyMD5 = SseGenericHeader + "-Customer-Key-MD5"
|
||||
|
||||
// SseCopyCustomerAlgorithm is the AWS SSE-C algorithm HTTP header key for CopyObject API.
|
||||
SseCopyCustomerAlgorithm = "X-Amz-Copy-Source-Server-Side-Encryption-Customer-Algorithm"
|
||||
// SseCopyCustomerKey is the AWS SSE-C encryption key HTTP header key for CopyObject API.
|
||||
SseCopyCustomerKey = "X-Amz-Copy-Source-Server-Side-Encryption-Customer-Key"
|
||||
// SseCopyCustomerKeyMD5 is the AWS SSE-C encryption key MD5 HTTP header key for CopyObject API.
|
||||
SseCopyCustomerKeyMD5 = "X-Amz-Copy-Source-Server-Side-Encryption-Customer-Key-MD5"
|
||||
)
|
||||
|
||||
// PBKDF creates a SSE-C key from the provided password and salt.
|
||||
// PBKDF is a password-based key derivation function
|
||||
// which can be used to derive a high-entropy cryptographic
|
||||
// key from a low-entropy password and a salt.
|
||||
type PBKDF func(password, salt []byte) ServerSide
|
||||
|
||||
// DefaultPBKDF is the default PBKDF. It uses Argon2id with the
|
||||
// recommended parameters from the RFC draft (1 pass, 64 MB memory, 4 threads).
|
||||
var DefaultPBKDF PBKDF = func(password, salt []byte) ServerSide {
|
||||
sse := ssec{}
|
||||
copy(sse[:], argon2.IDKey(password, salt, 1, 64*1024, 4, 32))
|
||||
return sse
|
||||
}
|
||||
|
||||
// Type is the server-side-encryption method. It represents one of
|
||||
// the following encryption methods:
|
||||
// - SSE-C: server-side-encryption with customer provided keys
|
||||
// - KMS: server-side-encryption with managed keys
|
||||
// - S3: server-side-encryption using S3 storage encryption
|
||||
type Type string
|
||||
|
||||
const (
|
||||
// SSEC represents server-side-encryption with customer provided keys
|
||||
SSEC Type = "SSE-C"
|
||||
// KMS represents server-side-encryption with managed keys
|
||||
KMS Type = "KMS"
|
||||
// S3 represents server-side-encryption using S3 storage encryption
|
||||
S3 Type = "S3"
|
||||
)
|
||||
|
||||
// ServerSide is a form of S3 server-side-encryption.
|
||||
type ServerSide interface {
|
||||
// Type returns the server-side-encryption method.
|
||||
Type() Type
|
||||
|
||||
// Marshal adds encryption headers to the provided HTTP headers.
|
||||
// It marks an HTTP request as server-side-encryption request
|
||||
// and inserts the required data into the headers.
|
||||
Marshal(h http.Header)
|
||||
}
|
||||
|
||||
// NewSSE returns a server-side-encryption using S3 storage encryption.
|
||||
// Using SSE-S3 the server will encrypt the object with server-managed keys.
|
||||
func NewSSE() ServerSide { return s3{} }
|
||||
|
||||
// NewSSEKMS returns a new server-side-encryption using SSE-KMS and the provided Key Id and context.
|
||||
func NewSSEKMS(keyID string, context interface{}) (ServerSide, error) {
|
||||
if context == nil {
|
||||
return kms{key: keyID, hasContext: false}, nil
|
||||
}
|
||||
serializedContext, err := json.Marshal(context)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return kms{key: keyID, context: serializedContext, hasContext: true}, nil
|
||||
}
|
||||
|
||||
// NewSSEC returns a new server-side-encryption using SSE-C and the provided key.
|
||||
// The key must be 32 bytes long.
|
||||
func NewSSEC(key []byte) (ServerSide, error) {
|
||||
if len(key) != 32 {
|
||||
return nil, errors.New("encrypt: SSE-C key must be 256 bit long")
|
||||
}
|
||||
sse := ssec{}
|
||||
copy(sse[:], key)
|
||||
return sse, nil
|
||||
}
|
||||
|
||||
// SSE transforms a SSE-C copy encryption into a SSE-C encryption.
|
||||
// It is the inverse of SSECopy(...).
|
||||
//
|
||||
// If the provided sse is no SSE-C copy encryption SSE returns
|
||||
// sse unmodified.
|
||||
func SSE(sse ServerSide) ServerSide {
|
||||
if sse == nil || sse.Type() != SSEC {
|
||||
return sse
|
||||
}
|
||||
if sse, ok := sse.(ssecCopy); ok {
|
||||
return ssec(sse)
|
||||
}
|
||||
return sse
|
||||
}
|
||||
|
||||
// SSECopy transforms a SSE-C encryption into a SSE-C copy
|
||||
// encryption. This is required for SSE-C key rotation or a SSE-C
|
||||
// copy where the source and the destination should be encrypted.
|
||||
//
|
||||
// If the provided sse is no SSE-C encryption SSECopy returns
|
||||
// sse unmodified.
|
||||
func SSECopy(sse ServerSide) ServerSide {
|
||||
if sse == nil || sse.Type() != SSEC {
|
||||
return sse
|
||||
}
|
||||
if sse, ok := sse.(ssec); ok {
|
||||
return ssecCopy(sse)
|
||||
}
|
||||
return sse
|
||||
}
|
||||
|
||||
type ssec [32]byte
|
||||
|
||||
func (s ssec) Type() Type { return SSEC }
|
||||
|
||||
func (s ssec) Marshal(h http.Header) {
|
||||
keyMD5 := md5.Sum(s[:])
|
||||
h.Set(SseCustomerAlgorithm, "AES256")
|
||||
h.Set(SseCustomerKey, base64.StdEncoding.EncodeToString(s[:]))
|
||||
h.Set(SseCustomerKeyMD5, base64.StdEncoding.EncodeToString(keyMD5[:]))
|
||||
}
|
||||
|
||||
type ssecCopy [32]byte
|
||||
|
||||
func (s ssecCopy) Type() Type { return SSEC }
|
||||
|
||||
func (s ssecCopy) Marshal(h http.Header) {
|
||||
keyMD5 := md5.Sum(s[:])
|
||||
h.Set(SseCopyCustomerAlgorithm, "AES256")
|
||||
h.Set(SseCopyCustomerKey, base64.StdEncoding.EncodeToString(s[:]))
|
||||
h.Set(SseCopyCustomerKeyMD5, base64.StdEncoding.EncodeToString(keyMD5[:]))
|
||||
}
|
||||
|
||||
type s3 struct{}
|
||||
|
||||
func (s s3) Type() Type { return S3 }
|
||||
|
||||
func (s s3) Marshal(h http.Header) { h.Set(SseGenericHeader, "AES256") }
|
||||
|
||||
type kms struct {
|
||||
key string
|
||||
context []byte
|
||||
hasContext bool
|
||||
}
|
||||
|
||||
func (s kms) Type() Type { return KMS }
|
||||
|
||||
func (s kms) Marshal(h http.Header) {
|
||||
h.Set(SseGenericHeader, "aws:kms")
|
||||
if s.key != "" {
|
||||
h.Set(SseKmsKeyID, s.key)
|
||||
}
|
||||
if s.hasContext {
|
||||
h.Set(SseEncryptionContext, base64.StdEncoding.EncodeToString(s.context))
|
||||
}
|
||||
}
|
||||
+542
@@ -0,0 +1,542 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2020 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
// Package lifecycle contains all the lifecycle related data types and marshallers.
|
||||
package lifecycle
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"encoding/xml"
|
||||
"errors"
|
||||
"time"
|
||||
)
|
||||
|
||||
var errMissingStorageClass = errors.New("storage-class cannot be empty")
|
||||
|
||||
// AbortIncompleteMultipartUpload structure, not supported yet on MinIO
|
||||
type AbortIncompleteMultipartUpload struct {
|
||||
XMLName xml.Name `xml:"AbortIncompleteMultipartUpload,omitempty" json:"-"`
|
||||
DaysAfterInitiation ExpirationDays `xml:"DaysAfterInitiation,omitempty" json:"DaysAfterInitiation,omitempty"`
|
||||
}
|
||||
|
||||
// IsDaysNull returns true if days field is null
|
||||
func (n AbortIncompleteMultipartUpload) IsDaysNull() bool {
|
||||
return n.DaysAfterInitiation == ExpirationDays(0)
|
||||
}
|
||||
|
||||
// MarshalXML if days after initiation is set to non-zero value
|
||||
func (n AbortIncompleteMultipartUpload) MarshalXML(e *xml.Encoder, start xml.StartElement) error {
|
||||
if n.IsDaysNull() {
|
||||
return nil
|
||||
}
|
||||
type abortIncompleteMultipartUploadWrapper AbortIncompleteMultipartUpload
|
||||
return e.EncodeElement(abortIncompleteMultipartUploadWrapper(n), start)
|
||||
}
|
||||
|
||||
// NoncurrentVersionExpiration - Specifies when noncurrent object versions expire.
|
||||
// Upon expiration, server permanently deletes the noncurrent object versions.
|
||||
// Set this lifecycle configuration action on a bucket that has versioning enabled
|
||||
// (or suspended) to request server delete noncurrent object versions at a
|
||||
// specific period in the object's lifetime.
|
||||
type NoncurrentVersionExpiration struct {
|
||||
XMLName xml.Name `xml:"NoncurrentVersionExpiration" json:"-"`
|
||||
NoncurrentDays ExpirationDays `xml:"NoncurrentDays,omitempty" json:"NoncurrentDays,omitempty"`
|
||||
NewerNoncurrentVersions int `xml:"NewerNoncurrentVersions,omitempty" json:"NewerNoncurrentVersions,omitempty"`
|
||||
}
|
||||
|
||||
// MarshalXML if n is non-empty, i.e has a non-zero NoncurrentDays or NewerNoncurrentVersions.
|
||||
func (n NoncurrentVersionExpiration) MarshalXML(e *xml.Encoder, start xml.StartElement) error {
|
||||
if n.isNull() {
|
||||
return nil
|
||||
}
|
||||
type noncurrentVersionExpirationWrapper NoncurrentVersionExpiration
|
||||
return e.EncodeElement(noncurrentVersionExpirationWrapper(n), start)
|
||||
}
|
||||
|
||||
// IsDaysNull returns true if days field is null
|
||||
func (n NoncurrentVersionExpiration) IsDaysNull() bool {
|
||||
return n.NoncurrentDays == ExpirationDays(0)
|
||||
}
|
||||
|
||||
func (n NoncurrentVersionExpiration) isNull() bool {
|
||||
return n.IsDaysNull() && n.NewerNoncurrentVersions == 0
|
||||
}
|
||||
|
||||
// NoncurrentVersionTransition structure, set this action to request server to
|
||||
// transition noncurrent object versions to different set storage classes
|
||||
// at a specific period in the object's lifetime.
|
||||
type NoncurrentVersionTransition struct {
|
||||
XMLName xml.Name `xml:"NoncurrentVersionTransition,omitempty" json:"-"`
|
||||
StorageClass string `xml:"StorageClass,omitempty" json:"StorageClass,omitempty"`
|
||||
NoncurrentDays ExpirationDays `xml:"NoncurrentDays" json:"NoncurrentDays"`
|
||||
NewerNoncurrentVersions int `xml:"NewerNoncurrentVersions,omitempty" json:"NewerNoncurrentVersions,omitempty"`
|
||||
}
|
||||
|
||||
// IsDaysNull returns true if days field is null
|
||||
func (n NoncurrentVersionTransition) IsDaysNull() bool {
|
||||
return n.NoncurrentDays == ExpirationDays(0)
|
||||
}
|
||||
|
||||
// IsStorageClassEmpty returns true if storage class field is empty
|
||||
func (n NoncurrentVersionTransition) IsStorageClassEmpty() bool {
|
||||
return n.StorageClass == ""
|
||||
}
|
||||
|
||||
func (n NoncurrentVersionTransition) isNull() bool {
|
||||
return n.StorageClass == ""
|
||||
}
|
||||
|
||||
// UnmarshalJSON implements NoncurrentVersionTransition JSONify
|
||||
func (n *NoncurrentVersionTransition) UnmarshalJSON(b []byte) error {
|
||||
type noncurrentVersionTransition NoncurrentVersionTransition
|
||||
var nt noncurrentVersionTransition
|
||||
err := json.Unmarshal(b, &nt)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if nt.StorageClass == "" {
|
||||
return errMissingStorageClass
|
||||
}
|
||||
*n = NoncurrentVersionTransition(nt)
|
||||
return nil
|
||||
}
|
||||
|
||||
// MarshalXML is extended to leave out
|
||||
// <NoncurrentVersionTransition></NoncurrentVersionTransition> tags
|
||||
func (n NoncurrentVersionTransition) MarshalXML(e *xml.Encoder, start xml.StartElement) error {
|
||||
if n.isNull() {
|
||||
return nil
|
||||
}
|
||||
type noncurrentVersionTransitionWrapper NoncurrentVersionTransition
|
||||
return e.EncodeElement(noncurrentVersionTransitionWrapper(n), start)
|
||||
}
|
||||
|
||||
// Tag structure key/value pair representing an object tag to apply lifecycle configuration
|
||||
type Tag struct {
|
||||
XMLName xml.Name `xml:"Tag,omitempty" json:"-"`
|
||||
Key string `xml:"Key,omitempty" json:"Key,omitempty"`
|
||||
Value string `xml:"Value,omitempty" json:"Value,omitempty"`
|
||||
}
|
||||
|
||||
// IsEmpty returns whether this tag is empty or not.
|
||||
func (tag Tag) IsEmpty() bool {
|
||||
return tag.Key == ""
|
||||
}
|
||||
|
||||
// Transition structure - transition details of lifecycle configuration
|
||||
type Transition struct {
|
||||
XMLName xml.Name `xml:"Transition" json:"-"`
|
||||
Date ExpirationDate `xml:"Date,omitempty" json:"Date,omitempty"`
|
||||
StorageClass string `xml:"StorageClass,omitempty" json:"StorageClass,omitempty"`
|
||||
Days ExpirationDays `xml:"Days" json:"Days"`
|
||||
}
|
||||
|
||||
// UnmarshalJSON returns an error if storage-class is empty.
|
||||
func (t *Transition) UnmarshalJSON(b []byte) error {
|
||||
type transition Transition
|
||||
var tr transition
|
||||
err := json.Unmarshal(b, &tr)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if tr.StorageClass == "" {
|
||||
return errMissingStorageClass
|
||||
}
|
||||
*t = Transition(tr)
|
||||
return nil
|
||||
}
|
||||
|
||||
// MarshalJSON customizes json encoding by omitting empty values
|
||||
func (t Transition) MarshalJSON() ([]byte, error) {
|
||||
if t.IsNull() {
|
||||
return nil, nil
|
||||
}
|
||||
type transition struct {
|
||||
Date *ExpirationDate `json:"Date,omitempty"`
|
||||
StorageClass string `json:"StorageClass,omitempty"`
|
||||
Days *ExpirationDays `json:"Days"`
|
||||
}
|
||||
|
||||
newt := transition{
|
||||
StorageClass: t.StorageClass,
|
||||
}
|
||||
|
||||
if !t.IsDateNull() {
|
||||
newt.Date = &t.Date
|
||||
} else {
|
||||
newt.Days = &t.Days
|
||||
}
|
||||
return json.Marshal(newt)
|
||||
}
|
||||
|
||||
// IsDaysNull returns true if days field is null
|
||||
func (t Transition) IsDaysNull() bool {
|
||||
return t.Days == ExpirationDays(0)
|
||||
}
|
||||
|
||||
// IsDateNull returns true if date field is null
|
||||
func (t Transition) IsDateNull() bool {
|
||||
return t.Date.Time.IsZero()
|
||||
}
|
||||
|
||||
// IsNull returns true if no storage-class is set.
|
||||
func (t Transition) IsNull() bool {
|
||||
return t.StorageClass == ""
|
||||
}
|
||||
|
||||
// MarshalXML is transition is non null
|
||||
func (t Transition) MarshalXML(en *xml.Encoder, startElement xml.StartElement) error {
|
||||
if t.IsNull() {
|
||||
return nil
|
||||
}
|
||||
type transitionWrapper Transition
|
||||
return en.EncodeElement(transitionWrapper(t), startElement)
|
||||
}
|
||||
|
||||
// And And Rule for LifecycleTag, to be used in LifecycleRuleFilter
|
||||
type And struct {
|
||||
XMLName xml.Name `xml:"And" json:"-"`
|
||||
Prefix string `xml:"Prefix" json:"Prefix,omitempty"`
|
||||
Tags []Tag `xml:"Tag" json:"Tags,omitempty"`
|
||||
ObjectSizeLessThan int64 `xml:"ObjectSizeLessThan,omitempty" json:"ObjectSizeLessThan,omitempty"`
|
||||
ObjectSizeGreaterThan int64 `xml:"ObjectSizeGreaterThan,omitempty" json:"ObjectSizeGreaterThan,omitempty"`
|
||||
}
|
||||
|
||||
// IsEmpty returns true if Tags field is null
|
||||
func (a And) IsEmpty() bool {
|
||||
return len(a.Tags) == 0 && a.Prefix == "" &&
|
||||
a.ObjectSizeLessThan == 0 && a.ObjectSizeGreaterThan == 0
|
||||
}
|
||||
|
||||
// Filter will be used in selecting rule(s) for lifecycle configuration
|
||||
type Filter struct {
|
||||
XMLName xml.Name `xml:"Filter" json:"-"`
|
||||
And And `xml:"And,omitempty" json:"And,omitempty"`
|
||||
Prefix string `xml:"Prefix,omitempty" json:"Prefix,omitempty"`
|
||||
Tag Tag `xml:"Tag,omitempty" json:"Tag,omitempty"`
|
||||
ObjectSizeLessThan int64 `xml:"ObjectSizeLessThan,omitempty" json:"ObjectSizeLessThan,omitempty"`
|
||||
ObjectSizeGreaterThan int64 `xml:"ObjectSizeGreaterThan,omitempty" json:"ObjectSizeGreaterThan,omitempty"`
|
||||
}
|
||||
|
||||
// IsNull returns true if all Filter fields are empty.
|
||||
func (f Filter) IsNull() bool {
|
||||
return f.Tag.IsEmpty() && f.And.IsEmpty() && f.Prefix == "" &&
|
||||
f.ObjectSizeLessThan == 0 && f.ObjectSizeGreaterThan == 0
|
||||
}
|
||||
|
||||
// MarshalJSON customizes json encoding by removing empty values.
|
||||
func (f Filter) MarshalJSON() ([]byte, error) {
|
||||
type filter struct {
|
||||
And *And `json:"And,omitempty"`
|
||||
Prefix string `json:"Prefix,omitempty"`
|
||||
Tag *Tag `json:"Tag,omitempty"`
|
||||
ObjectSizeLessThan int64 `json:"ObjectSizeLessThan,omitempty"`
|
||||
ObjectSizeGreaterThan int64 `json:"ObjectSizeGreaterThan,omitempty"`
|
||||
}
|
||||
|
||||
newf := filter{
|
||||
Prefix: f.Prefix,
|
||||
}
|
||||
if !f.Tag.IsEmpty() {
|
||||
newf.Tag = &f.Tag
|
||||
}
|
||||
if !f.And.IsEmpty() {
|
||||
newf.And = &f.And
|
||||
}
|
||||
newf.ObjectSizeLessThan = f.ObjectSizeLessThan
|
||||
newf.ObjectSizeGreaterThan = f.ObjectSizeGreaterThan
|
||||
return json.Marshal(newf)
|
||||
}
|
||||
|
||||
// MarshalXML - produces the xml representation of the Filter struct
|
||||
// only one of Prefix, And and Tag should be present in the output.
|
||||
func (f Filter) MarshalXML(e *xml.Encoder, start xml.StartElement) error {
|
||||
if err := e.EncodeToken(start); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
switch {
|
||||
case !f.And.IsEmpty():
|
||||
if err := e.EncodeElement(f.And, xml.StartElement{Name: xml.Name{Local: "And"}}); err != nil {
|
||||
return err
|
||||
}
|
||||
case !f.Tag.IsEmpty():
|
||||
if err := e.EncodeElement(f.Tag, xml.StartElement{Name: xml.Name{Local: "Tag"}}); err != nil {
|
||||
return err
|
||||
}
|
||||
default:
|
||||
if f.ObjectSizeLessThan > 0 {
|
||||
if err := e.EncodeElement(f.ObjectSizeLessThan, xml.StartElement{Name: xml.Name{Local: "ObjectSizeLessThan"}}); err != nil {
|
||||
return err
|
||||
}
|
||||
break
|
||||
}
|
||||
if f.ObjectSizeGreaterThan > 0 {
|
||||
if err := e.EncodeElement(f.ObjectSizeGreaterThan, xml.StartElement{Name: xml.Name{Local: "ObjectSizeGreaterThan"}}); err != nil {
|
||||
return err
|
||||
}
|
||||
break
|
||||
}
|
||||
// Print empty Prefix field only when everything else is empty
|
||||
if err := e.EncodeElement(f.Prefix, xml.StartElement{Name: xml.Name{Local: "Prefix"}}); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return e.EncodeToken(xml.EndElement{Name: start.Name})
|
||||
}
|
||||
|
||||
// ExpirationDays is a type alias to unmarshal Days in Expiration
|
||||
type ExpirationDays int
|
||||
|
||||
// MarshalXML encodes number of days to expire if it is non-zero and
|
||||
// encodes empty string otherwise
|
||||
func (eDays ExpirationDays) MarshalXML(e *xml.Encoder, startElement xml.StartElement) error {
|
||||
if eDays == 0 {
|
||||
return nil
|
||||
}
|
||||
return e.EncodeElement(int(eDays), startElement)
|
||||
}
|
||||
|
||||
// ExpirationDate is a embedded type containing time.Time to unmarshal
|
||||
// Date in Expiration
|
||||
type ExpirationDate struct {
|
||||
time.Time
|
||||
}
|
||||
|
||||
// MarshalXML encodes expiration date if it is non-zero and encodes
|
||||
// empty string otherwise
|
||||
func (eDate ExpirationDate) MarshalXML(e *xml.Encoder, startElement xml.StartElement) error {
|
||||
if eDate.Time.IsZero() {
|
||||
return nil
|
||||
}
|
||||
return e.EncodeElement(eDate.Format(time.RFC3339), startElement)
|
||||
}
|
||||
|
||||
// ExpireDeleteMarker represents value of ExpiredObjectDeleteMarker field in Expiration XML element.
|
||||
type ExpireDeleteMarker ExpirationBoolean
|
||||
|
||||
// IsEnabled returns true if the auto delete-marker expiration is enabled
|
||||
func (e ExpireDeleteMarker) IsEnabled() bool {
|
||||
return bool(e)
|
||||
}
|
||||
|
||||
// ExpirationBoolean represents an XML version of 'bool' type
|
||||
type ExpirationBoolean bool
|
||||
|
||||
// MarshalXML encodes delete marker boolean into an XML form.
|
||||
func (b ExpirationBoolean) MarshalXML(e *xml.Encoder, startElement xml.StartElement) error {
|
||||
if !b {
|
||||
return nil
|
||||
}
|
||||
type booleanWrapper ExpirationBoolean
|
||||
return e.EncodeElement(booleanWrapper(b), startElement)
|
||||
}
|
||||
|
||||
// IsEnabled returns true if the expiration boolean is enabled
|
||||
func (b ExpirationBoolean) IsEnabled() bool {
|
||||
return bool(b)
|
||||
}
|
||||
|
||||
// Expiration structure - expiration details of lifecycle configuration
|
||||
type Expiration struct {
|
||||
XMLName xml.Name `xml:"Expiration,omitempty" json:"-"`
|
||||
Date ExpirationDate `xml:"Date,omitempty" json:"Date,omitempty"`
|
||||
Days ExpirationDays `xml:"Days,omitempty" json:"Days,omitempty"`
|
||||
DeleteMarker ExpireDeleteMarker `xml:"ExpiredObjectDeleteMarker,omitempty" json:"ExpiredObjectDeleteMarker,omitempty"`
|
||||
DeleteAll ExpirationBoolean `xml:"ExpiredObjectAllVersions,omitempty" json:"ExpiredObjectAllVersions,omitempty"`
|
||||
}
|
||||
|
||||
// MarshalJSON customizes json encoding by removing empty day/date specification.
|
||||
func (e Expiration) MarshalJSON() ([]byte, error) {
|
||||
type expiration struct {
|
||||
Date *ExpirationDate `json:"Date,omitempty"`
|
||||
Days *ExpirationDays `json:"Days,omitempty"`
|
||||
DeleteMarker ExpireDeleteMarker `json:"ExpiredObjectDeleteMarker,omitempty"`
|
||||
DeleteAll ExpirationBoolean `json:"ExpiredObjectAllVersions,omitempty"`
|
||||
}
|
||||
|
||||
newexp := expiration{
|
||||
DeleteMarker: e.DeleteMarker,
|
||||
DeleteAll: e.DeleteAll,
|
||||
}
|
||||
if !e.IsDaysNull() {
|
||||
newexp.Days = &e.Days
|
||||
}
|
||||
if !e.IsDateNull() {
|
||||
newexp.Date = &e.Date
|
||||
}
|
||||
return json.Marshal(newexp)
|
||||
}
|
||||
|
||||
// IsDaysNull returns true if days field is null
|
||||
func (e Expiration) IsDaysNull() bool {
|
||||
return e.Days == ExpirationDays(0)
|
||||
}
|
||||
|
||||
// IsDateNull returns true if date field is null
|
||||
func (e Expiration) IsDateNull() bool {
|
||||
return e.Date.Time.IsZero()
|
||||
}
|
||||
|
||||
// IsDeleteMarkerExpirationEnabled returns true if the auto-expiration of delete marker is enabled
|
||||
func (e Expiration) IsDeleteMarkerExpirationEnabled() bool {
|
||||
return e.DeleteMarker.IsEnabled()
|
||||
}
|
||||
|
||||
// IsNull returns true if both date and days fields are null
|
||||
func (e Expiration) IsNull() bool {
|
||||
return e.IsDaysNull() && e.IsDateNull() && !e.IsDeleteMarkerExpirationEnabled() && !e.DeleteAll.IsEnabled()
|
||||
}
|
||||
|
||||
// MarshalXML is expiration is non null
|
||||
func (e Expiration) MarshalXML(en *xml.Encoder, startElement xml.StartElement) error {
|
||||
if e.IsNull() {
|
||||
return nil
|
||||
}
|
||||
type expirationWrapper Expiration
|
||||
return en.EncodeElement(expirationWrapper(e), startElement)
|
||||
}
|
||||
|
||||
// DelMarkerExpiration represents DelMarkerExpiration actions element in an ILM policy
|
||||
type DelMarkerExpiration struct {
|
||||
XMLName xml.Name `xml:"DelMarkerExpiration" json:"-"`
|
||||
Days int `xml:"Days,omitempty" json:"Days,omitempty"`
|
||||
}
|
||||
|
||||
// IsNull returns true if Days isn't specified and false otherwise.
|
||||
func (de DelMarkerExpiration) IsNull() bool {
|
||||
return de.Days == 0
|
||||
}
|
||||
|
||||
// MarshalXML avoids serializing an empty DelMarkerExpiration element
|
||||
func (de DelMarkerExpiration) MarshalXML(enc *xml.Encoder, start xml.StartElement) error {
|
||||
if de.IsNull() {
|
||||
return nil
|
||||
}
|
||||
type delMarkerExp DelMarkerExpiration
|
||||
return enc.EncodeElement(delMarkerExp(de), start)
|
||||
}
|
||||
|
||||
// AllVersionsExpiration represents AllVersionsExpiration actions element in an ILM policy
|
||||
type AllVersionsExpiration struct {
|
||||
XMLName xml.Name `xml:"AllVersionsExpiration" json:"-"`
|
||||
Days int `xml:"Days,omitempty" json:"Days,omitempty"`
|
||||
DeleteMarker ExpireDeleteMarker `xml:"DeleteMarker,omitempty" json:"DeleteMarker,omitempty"`
|
||||
}
|
||||
|
||||
// IsNull returns true if days field is 0
|
||||
func (e AllVersionsExpiration) IsNull() bool {
|
||||
return e.Days == 0
|
||||
}
|
||||
|
||||
// MarshalXML satisfies xml.Marshaler to provide custom encoding
|
||||
func (e AllVersionsExpiration) MarshalXML(enc *xml.Encoder, start xml.StartElement) error {
|
||||
if e.IsNull() {
|
||||
return nil
|
||||
}
|
||||
type allVersionsExp AllVersionsExpiration
|
||||
return enc.EncodeElement(allVersionsExp(e), start)
|
||||
}
|
||||
|
||||
// MarshalJSON customizes json encoding by omitting empty values
|
||||
func (r Rule) MarshalJSON() ([]byte, error) {
|
||||
type rule struct {
|
||||
AbortIncompleteMultipartUpload *AbortIncompleteMultipartUpload `json:"AbortIncompleteMultipartUpload,omitempty"`
|
||||
Expiration *Expiration `json:"Expiration,omitempty"`
|
||||
DelMarkerExpiration *DelMarkerExpiration `json:"DelMarkerExpiration,omitempty"`
|
||||
AllVersionsExpiration *AllVersionsExpiration `json:"AllVersionsExpiration,omitempty"`
|
||||
ID string `json:"ID"`
|
||||
RuleFilter *Filter `json:"Filter,omitempty"`
|
||||
NoncurrentVersionExpiration *NoncurrentVersionExpiration `json:"NoncurrentVersionExpiration,omitempty"`
|
||||
NoncurrentVersionTransition *NoncurrentVersionTransition `json:"NoncurrentVersionTransition,omitempty"`
|
||||
Prefix string `json:"Prefix,omitempty"`
|
||||
Status string `json:"Status"`
|
||||
Transition *Transition `json:"Transition,omitempty"`
|
||||
}
|
||||
newr := rule{
|
||||
Prefix: r.Prefix,
|
||||
Status: r.Status,
|
||||
ID: r.ID,
|
||||
}
|
||||
|
||||
if !r.RuleFilter.IsNull() {
|
||||
newr.RuleFilter = &r.RuleFilter
|
||||
}
|
||||
if !r.AbortIncompleteMultipartUpload.IsDaysNull() {
|
||||
newr.AbortIncompleteMultipartUpload = &r.AbortIncompleteMultipartUpload
|
||||
}
|
||||
if !r.Expiration.IsNull() {
|
||||
newr.Expiration = &r.Expiration
|
||||
}
|
||||
if !r.DelMarkerExpiration.IsNull() {
|
||||
newr.DelMarkerExpiration = &r.DelMarkerExpiration
|
||||
}
|
||||
if !r.Transition.IsNull() {
|
||||
newr.Transition = &r.Transition
|
||||
}
|
||||
if !r.NoncurrentVersionExpiration.isNull() {
|
||||
newr.NoncurrentVersionExpiration = &r.NoncurrentVersionExpiration
|
||||
}
|
||||
if !r.NoncurrentVersionTransition.isNull() {
|
||||
newr.NoncurrentVersionTransition = &r.NoncurrentVersionTransition
|
||||
}
|
||||
if !r.AllVersionsExpiration.IsNull() {
|
||||
newr.AllVersionsExpiration = &r.AllVersionsExpiration
|
||||
}
|
||||
|
||||
return json.Marshal(newr)
|
||||
}
|
||||
|
||||
// Rule represents a single rule in lifecycle configuration
|
||||
type Rule struct {
|
||||
XMLName xml.Name `xml:"Rule,omitempty" json:"-"`
|
||||
AbortIncompleteMultipartUpload AbortIncompleteMultipartUpload `xml:"AbortIncompleteMultipartUpload,omitempty" json:"AbortIncompleteMultipartUpload,omitempty"`
|
||||
Expiration Expiration `xml:"Expiration,omitempty" json:"Expiration,omitempty"`
|
||||
DelMarkerExpiration DelMarkerExpiration `xml:"DelMarkerExpiration,omitempty" json:"DelMarkerExpiration,omitempty"`
|
||||
AllVersionsExpiration AllVersionsExpiration `xml:"AllVersionsExpiration,omitempty" json:"AllVersionsExpiration,omitempty"`
|
||||
ID string `xml:"ID" json:"ID"`
|
||||
RuleFilter Filter `xml:"Filter,omitempty" json:"Filter,omitempty"`
|
||||
NoncurrentVersionExpiration NoncurrentVersionExpiration `xml:"NoncurrentVersionExpiration,omitempty" json:"NoncurrentVersionExpiration,omitempty"`
|
||||
NoncurrentVersionTransition NoncurrentVersionTransition `xml:"NoncurrentVersionTransition,omitempty" json:"NoncurrentVersionTransition,omitempty"`
|
||||
Prefix string `xml:"Prefix,omitempty" json:"Prefix,omitempty"`
|
||||
Status string `xml:"Status" json:"Status"`
|
||||
Transition Transition `xml:"Transition,omitempty" json:"Transition,omitempty"`
|
||||
}
|
||||
|
||||
// Configuration is a collection of Rule objects.
|
||||
type Configuration struct {
|
||||
XMLName xml.Name `xml:"LifecycleConfiguration,omitempty" json:"-"`
|
||||
Rules []Rule `xml:"Rule"`
|
||||
}
|
||||
|
||||
// Empty check if lifecycle configuration is empty
|
||||
func (c *Configuration) Empty() bool {
|
||||
if c == nil {
|
||||
return true
|
||||
}
|
||||
return len(c.Rules) == 0
|
||||
}
|
||||
|
||||
// NewConfiguration initializes a fresh lifecycle configuration
|
||||
// for manipulation, such as setting and removing lifecycle rules
|
||||
// and filters.
|
||||
func NewConfiguration() *Configuration {
|
||||
return &Configuration{}
|
||||
}
|
||||
+78
@@ -0,0 +1,78 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2017-2020 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package notification
|
||||
|
||||
// Indentity represents the user id, this is a compliance field.
|
||||
type identity struct {
|
||||
PrincipalID string `json:"principalId"`
|
||||
}
|
||||
|
||||
// event bucket metadata.
|
||||
type bucketMeta struct {
|
||||
Name string `json:"name"`
|
||||
OwnerIdentity identity `json:"ownerIdentity"`
|
||||
ARN string `json:"arn"`
|
||||
}
|
||||
|
||||
// event object metadata.
|
||||
type objectMeta struct {
|
||||
Key string `json:"key"`
|
||||
Size int64 `json:"size,omitempty"`
|
||||
ETag string `json:"eTag,omitempty"`
|
||||
ContentType string `json:"contentType,omitempty"`
|
||||
UserMetadata map[string]string `json:"userMetadata,omitempty"`
|
||||
VersionID string `json:"versionId,omitempty"`
|
||||
Sequencer string `json:"sequencer"`
|
||||
}
|
||||
|
||||
// event server specific metadata.
|
||||
type eventMeta struct {
|
||||
SchemaVersion string `json:"s3SchemaVersion"`
|
||||
ConfigurationID string `json:"configurationId"`
|
||||
Bucket bucketMeta `json:"bucket"`
|
||||
Object objectMeta `json:"object"`
|
||||
}
|
||||
|
||||
// sourceInfo represents information on the client that
|
||||
// triggered the event notification.
|
||||
type sourceInfo struct {
|
||||
Host string `json:"host"`
|
||||
Port string `json:"port"`
|
||||
UserAgent string `json:"userAgent"`
|
||||
}
|
||||
|
||||
// Event represents an Amazon an S3 bucket notification event.
|
||||
type Event struct {
|
||||
EventVersion string `json:"eventVersion"`
|
||||
EventSource string `json:"eventSource"`
|
||||
AwsRegion string `json:"awsRegion"`
|
||||
EventTime string `json:"eventTime"`
|
||||
EventName string `json:"eventName"`
|
||||
UserIdentity identity `json:"userIdentity"`
|
||||
RequestParameters map[string]string `json:"requestParameters"`
|
||||
ResponseElements map[string]string `json:"responseElements"`
|
||||
S3 eventMeta `json:"s3"`
|
||||
Source sourceInfo `json:"source"`
|
||||
}
|
||||
|
||||
// Info - represents the collection of notification events, additionally
|
||||
// also reports errors if any while listening on bucket notifications.
|
||||
type Info struct {
|
||||
Records []Event
|
||||
Err error
|
||||
}
|
||||
Generated
Vendored
+441
@@ -0,0 +1,441 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2020 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package notification
|
||||
|
||||
import (
|
||||
"encoding/xml"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/minio/minio-go/v7/pkg/set"
|
||||
)
|
||||
|
||||
// EventType is a S3 notification event associated to the bucket notification configuration
|
||||
type EventType string
|
||||
|
||||
// The role of all event types are described in :
|
||||
//
|
||||
// http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html#notification-how-to-event-types-and-destinations
|
||||
const (
|
||||
ObjectCreatedAll EventType = "s3:ObjectCreated:*"
|
||||
ObjectCreatedPut EventType = "s3:ObjectCreated:Put"
|
||||
ObjectCreatedPost EventType = "s3:ObjectCreated:Post"
|
||||
ObjectCreatedCopy EventType = "s3:ObjectCreated:Copy"
|
||||
ObjectCreatedDeleteTagging EventType = "s3:ObjectCreated:DeleteTagging"
|
||||
ObjectCreatedCompleteMultipartUpload EventType = "s3:ObjectCreated:CompleteMultipartUpload"
|
||||
ObjectCreatedPutLegalHold EventType = "s3:ObjectCreated:PutLegalHold"
|
||||
ObjectCreatedPutRetention EventType = "s3:ObjectCreated:PutRetention"
|
||||
ObjectCreatedPutTagging EventType = "s3:ObjectCreated:PutTagging"
|
||||
ObjectAccessedGet EventType = "s3:ObjectAccessed:Get"
|
||||
ObjectAccessedHead EventType = "s3:ObjectAccessed:Head"
|
||||
ObjectAccessedGetRetention EventType = "s3:ObjectAccessed:GetRetention"
|
||||
ObjectAccessedGetLegalHold EventType = "s3:ObjectAccessed:GetLegalHold"
|
||||
ObjectAccessedAll EventType = "s3:ObjectAccessed:*"
|
||||
ObjectRemovedAll EventType = "s3:ObjectRemoved:*"
|
||||
ObjectRemovedDelete EventType = "s3:ObjectRemoved:Delete"
|
||||
ObjectRemovedDeleteMarkerCreated EventType = "s3:ObjectRemoved:DeleteMarkerCreated"
|
||||
ILMDelMarkerExpirationDelete EventType = "s3:LifecycleDelMarkerExpiration:Delete"
|
||||
ObjectReducedRedundancyLostObject EventType = "s3:ReducedRedundancyLostObject"
|
||||
ObjectTransitionAll EventType = "s3:ObjectTransition:*"
|
||||
ObjectTransitionFailed EventType = "s3:ObjectTransition:Failed"
|
||||
ObjectTransitionComplete EventType = "s3:ObjectTransition:Complete"
|
||||
ObjectTransitionPost EventType = "s3:ObjectRestore:Post"
|
||||
ObjectTransitionCompleted EventType = "s3:ObjectRestore:Completed"
|
||||
ObjectReplicationAll EventType = "s3:Replication:*"
|
||||
ObjectReplicationOperationCompletedReplication EventType = "s3:Replication:OperationCompletedReplication"
|
||||
ObjectReplicationOperationFailedReplication EventType = "s3:Replication:OperationFailedReplication"
|
||||
ObjectReplicationOperationMissedThreshold EventType = "s3:Replication:OperationMissedThreshold"
|
||||
ObjectReplicationOperationNotTracked EventType = "s3:Replication:OperationNotTracked"
|
||||
ObjectReplicationOperationReplicatedAfterThreshold EventType = "s3:Replication:OperationReplicatedAfterThreshold"
|
||||
ObjectScannerManyVersions EventType = "s3:Scanner:ManyVersions"
|
||||
ObjectScannerBigPrefix EventType = "s3:Scanner:BigPrefix"
|
||||
ObjectScannerAll EventType = "s3:Scanner:*"
|
||||
BucketCreatedAll EventType = "s3:BucketCreated:*"
|
||||
BucketRemovedAll EventType = "s3:BucketRemoved:*"
|
||||
)
|
||||
|
||||
// FilterRule - child of S3Key, a tag in the notification xml which
|
||||
// carries suffix/prefix filters
|
||||
type FilterRule struct {
|
||||
Name string `xml:"Name"`
|
||||
Value string `xml:"Value"`
|
||||
}
|
||||
|
||||
// S3Key - child of Filter, a tag in the notification xml which
|
||||
// carries suffix/prefix filters
|
||||
type S3Key struct {
|
||||
FilterRules []FilterRule `xml:"FilterRule,omitempty"`
|
||||
}
|
||||
|
||||
// Filter - a tag in the notification xml structure which carries
|
||||
// suffix/prefix filters
|
||||
type Filter struct {
|
||||
S3Key S3Key `xml:"S3Key,omitempty"`
|
||||
}
|
||||
|
||||
// Arn - holds ARN information that will be sent to the web service,
|
||||
// ARN desciption can be found in http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html
|
||||
type Arn struct {
|
||||
Partition string
|
||||
Service string
|
||||
Region string
|
||||
AccountID string
|
||||
Resource string
|
||||
}
|
||||
|
||||
// NewArn creates new ARN based on the given partition, service, region, account id and resource
|
||||
func NewArn(partition, service, region, accountID, resource string) Arn {
|
||||
return Arn{
|
||||
Partition: partition,
|
||||
Service: service,
|
||||
Region: region,
|
||||
AccountID: accountID,
|
||||
Resource: resource,
|
||||
}
|
||||
}
|
||||
|
||||
var (
|
||||
// ErrInvalidArnPrefix is returned when ARN string format does not start with 'arn'
|
||||
ErrInvalidArnPrefix = errors.New("invalid ARN format, must start with 'arn:'")
|
||||
// ErrInvalidArnFormat is returned when ARN string format is not valid
|
||||
ErrInvalidArnFormat = errors.New("invalid ARN format, must be 'arn:<partition>:<service>:<region>:<accountID>:<resource>'")
|
||||
)
|
||||
|
||||
// NewArnFromString parses string representation of ARN into Arn object.
|
||||
// Returns an error if the string format is incorrect.
|
||||
func NewArnFromString(arn string) (Arn, error) {
|
||||
parts := strings.Split(arn, ":")
|
||||
if len(parts) != 6 {
|
||||
return Arn{}, ErrInvalidArnFormat
|
||||
}
|
||||
if parts[0] != "arn" {
|
||||
return Arn{}, ErrInvalidArnPrefix
|
||||
}
|
||||
|
||||
return NewArn(parts[1], parts[2], parts[3], parts[4], parts[5]), nil
|
||||
}
|
||||
|
||||
// String returns the string format of the ARN
|
||||
func (arn Arn) String() string {
|
||||
return "arn:" + arn.Partition + ":" + arn.Service + ":" + arn.Region + ":" + arn.AccountID + ":" + arn.Resource
|
||||
}
|
||||
|
||||
// Config - represents one single notification configuration
|
||||
// such as topic, queue or lambda configuration.
|
||||
type Config struct {
|
||||
ID string `xml:"Id,omitempty"`
|
||||
Arn Arn `xml:"-"`
|
||||
Events []EventType `xml:"Event"`
|
||||
Filter *Filter `xml:"Filter,omitempty"`
|
||||
}
|
||||
|
||||
// NewConfig creates one notification config and sets the given ARN
|
||||
func NewConfig(arn Arn) Config {
|
||||
return Config{Arn: arn, Filter: &Filter{}}
|
||||
}
|
||||
|
||||
// AddEvents adds one event to the current notification config
|
||||
func (t *Config) AddEvents(events ...EventType) {
|
||||
t.Events = append(t.Events, events...)
|
||||
}
|
||||
|
||||
// AddFilterSuffix sets the suffix configuration to the current notification config
|
||||
func (t *Config) AddFilterSuffix(suffix string) {
|
||||
if t.Filter == nil {
|
||||
t.Filter = &Filter{}
|
||||
}
|
||||
newFilterRule := FilterRule{Name: "suffix", Value: suffix}
|
||||
// Replace any suffix rule if existing and add to the list otherwise
|
||||
for index := range t.Filter.S3Key.FilterRules {
|
||||
if t.Filter.S3Key.FilterRules[index].Name == "suffix" {
|
||||
t.Filter.S3Key.FilterRules[index] = newFilterRule
|
||||
return
|
||||
}
|
||||
}
|
||||
t.Filter.S3Key.FilterRules = append(t.Filter.S3Key.FilterRules, newFilterRule)
|
||||
}
|
||||
|
||||
// AddFilterPrefix sets the prefix configuration to the current notification config
|
||||
func (t *Config) AddFilterPrefix(prefix string) {
|
||||
if t.Filter == nil {
|
||||
t.Filter = &Filter{}
|
||||
}
|
||||
newFilterRule := FilterRule{Name: "prefix", Value: prefix}
|
||||
// Replace any prefix rule if existing and add to the list otherwise
|
||||
for index := range t.Filter.S3Key.FilterRules {
|
||||
if t.Filter.S3Key.FilterRules[index].Name == "prefix" {
|
||||
t.Filter.S3Key.FilterRules[index] = newFilterRule
|
||||
return
|
||||
}
|
||||
}
|
||||
t.Filter.S3Key.FilterRules = append(t.Filter.S3Key.FilterRules, newFilterRule)
|
||||
}
|
||||
|
||||
// EqualEventTypeList tells whether a and b contain the same events
|
||||
func EqualEventTypeList(a, b []EventType) bool {
|
||||
if len(a) != len(b) {
|
||||
return false
|
||||
}
|
||||
setA := set.NewStringSet()
|
||||
for _, i := range a {
|
||||
setA.Add(string(i))
|
||||
}
|
||||
|
||||
setB := set.NewStringSet()
|
||||
for _, i := range b {
|
||||
setB.Add(string(i))
|
||||
}
|
||||
|
||||
return setA.Difference(setB).IsEmpty()
|
||||
}
|
||||
|
||||
// EqualFilterRuleList tells whether a and b contain the same filters
|
||||
func EqualFilterRuleList(a, b []FilterRule) bool {
|
||||
if len(a) != len(b) {
|
||||
return false
|
||||
}
|
||||
|
||||
setA := set.NewStringSet()
|
||||
for _, i := range a {
|
||||
setA.Add(fmt.Sprintf("%s-%s", i.Name, i.Value))
|
||||
}
|
||||
|
||||
setB := set.NewStringSet()
|
||||
for _, i := range b {
|
||||
setB.Add(fmt.Sprintf("%s-%s", i.Name, i.Value))
|
||||
}
|
||||
|
||||
return setA.Difference(setB).IsEmpty()
|
||||
}
|
||||
|
||||
// Equal returns whether this `Config` is equal to another defined by the passed parameters
|
||||
func (t *Config) Equal(events []EventType, prefix, suffix string) bool {
|
||||
if t == nil {
|
||||
return false
|
||||
}
|
||||
|
||||
// Compare events
|
||||
passEvents := EqualEventTypeList(t.Events, events)
|
||||
|
||||
// Compare filters
|
||||
var newFilterRules []FilterRule
|
||||
if prefix != "" {
|
||||
newFilterRules = append(newFilterRules, FilterRule{Name: "prefix", Value: prefix})
|
||||
}
|
||||
if suffix != "" {
|
||||
newFilterRules = append(newFilterRules, FilterRule{Name: "suffix", Value: suffix})
|
||||
}
|
||||
|
||||
var currentFilterRules []FilterRule
|
||||
if t.Filter != nil {
|
||||
currentFilterRules = t.Filter.S3Key.FilterRules
|
||||
}
|
||||
|
||||
passFilters := EqualFilterRuleList(currentFilterRules, newFilterRules)
|
||||
return passEvents && passFilters
|
||||
}
|
||||
|
||||
// TopicConfig carries one single topic notification configuration
|
||||
type TopicConfig struct {
|
||||
Config
|
||||
Topic string `xml:"Topic"`
|
||||
}
|
||||
|
||||
// QueueConfig carries one single queue notification configuration
|
||||
type QueueConfig struct {
|
||||
Config
|
||||
Queue string `xml:"Queue"`
|
||||
}
|
||||
|
||||
// LambdaConfig carries one single cloudfunction notification configuration
|
||||
type LambdaConfig struct {
|
||||
Config
|
||||
Lambda string `xml:"CloudFunction"`
|
||||
}
|
||||
|
||||
// Configuration - the struct that represents the whole XML to be sent to the web service
|
||||
type Configuration struct {
|
||||
XMLName xml.Name `xml:"NotificationConfiguration"`
|
||||
LambdaConfigs []LambdaConfig `xml:"CloudFunctionConfiguration"`
|
||||
TopicConfigs []TopicConfig `xml:"TopicConfiguration"`
|
||||
QueueConfigs []QueueConfig `xml:"QueueConfiguration"`
|
||||
}
|
||||
|
||||
// AddTopic adds a given topic config to the general bucket notification config
|
||||
func (b *Configuration) AddTopic(topicConfig Config) bool {
|
||||
newTopicConfig := TopicConfig{Config: topicConfig, Topic: topicConfig.Arn.String()}
|
||||
for _, n := range b.TopicConfigs {
|
||||
// If new config matches existing one
|
||||
if n.Topic == newTopicConfig.Arn.String() && newTopicConfig.Filter == n.Filter {
|
||||
|
||||
existingConfig := set.NewStringSet()
|
||||
for _, v := range n.Events {
|
||||
existingConfig.Add(string(v))
|
||||
}
|
||||
|
||||
newConfig := set.NewStringSet()
|
||||
for _, v := range topicConfig.Events {
|
||||
newConfig.Add(string(v))
|
||||
}
|
||||
|
||||
if !newConfig.Intersection(existingConfig).IsEmpty() {
|
||||
return false
|
||||
}
|
||||
}
|
||||
}
|
||||
b.TopicConfigs = append(b.TopicConfigs, newTopicConfig)
|
||||
return true
|
||||
}
|
||||
|
||||
// AddQueue adds a given queue config to the general bucket notification config
|
||||
func (b *Configuration) AddQueue(queueConfig Config) bool {
|
||||
newQueueConfig := QueueConfig{Config: queueConfig, Queue: queueConfig.Arn.String()}
|
||||
for _, n := range b.QueueConfigs {
|
||||
if n.Queue == newQueueConfig.Arn.String() && newQueueConfig.Filter == n.Filter {
|
||||
|
||||
existingConfig := set.NewStringSet()
|
||||
for _, v := range n.Events {
|
||||
existingConfig.Add(string(v))
|
||||
}
|
||||
|
||||
newConfig := set.NewStringSet()
|
||||
for _, v := range queueConfig.Events {
|
||||
newConfig.Add(string(v))
|
||||
}
|
||||
|
||||
if !newConfig.Intersection(existingConfig).IsEmpty() {
|
||||
return false
|
||||
}
|
||||
}
|
||||
}
|
||||
b.QueueConfigs = append(b.QueueConfigs, newQueueConfig)
|
||||
return true
|
||||
}
|
||||
|
||||
// AddLambda adds a given lambda config to the general bucket notification config
|
||||
func (b *Configuration) AddLambda(lambdaConfig Config) bool {
|
||||
newLambdaConfig := LambdaConfig{Config: lambdaConfig, Lambda: lambdaConfig.Arn.String()}
|
||||
for _, n := range b.LambdaConfigs {
|
||||
if n.Lambda == newLambdaConfig.Arn.String() && newLambdaConfig.Filter == n.Filter {
|
||||
|
||||
existingConfig := set.NewStringSet()
|
||||
for _, v := range n.Events {
|
||||
existingConfig.Add(string(v))
|
||||
}
|
||||
|
||||
newConfig := set.NewStringSet()
|
||||
for _, v := range lambdaConfig.Events {
|
||||
newConfig.Add(string(v))
|
||||
}
|
||||
|
||||
if !newConfig.Intersection(existingConfig).IsEmpty() {
|
||||
return false
|
||||
}
|
||||
}
|
||||
}
|
||||
b.LambdaConfigs = append(b.LambdaConfigs, newLambdaConfig)
|
||||
return true
|
||||
}
|
||||
|
||||
// RemoveTopicByArn removes all topic configurations that match the exact specified ARN
|
||||
func (b *Configuration) RemoveTopicByArn(arn Arn) {
|
||||
var topics []TopicConfig
|
||||
for _, topic := range b.TopicConfigs {
|
||||
if topic.Topic != arn.String() {
|
||||
topics = append(topics, topic)
|
||||
}
|
||||
}
|
||||
b.TopicConfigs = topics
|
||||
}
|
||||
|
||||
// ErrNoConfigMatch is returned when a notification configuration (sqs,sns,lambda) is not found when trying to delete
|
||||
var ErrNoConfigMatch = errors.New("no notification configuration matched")
|
||||
|
||||
// RemoveTopicByArnEventsPrefixSuffix removes a topic configuration that match the exact specified ARN, events, prefix and suffix
|
||||
func (b *Configuration) RemoveTopicByArnEventsPrefixSuffix(arn Arn, events []EventType, prefix, suffix string) error {
|
||||
removeIndex := -1
|
||||
for i, v := range b.TopicConfigs {
|
||||
// if it matches events and filters, mark the index for deletion
|
||||
if v.Topic == arn.String() && v.Config.Equal(events, prefix, suffix) {
|
||||
removeIndex = i
|
||||
break // since we have at most one matching config
|
||||
}
|
||||
}
|
||||
if removeIndex >= 0 {
|
||||
b.TopicConfigs = append(b.TopicConfigs[:removeIndex], b.TopicConfigs[removeIndex+1:]...)
|
||||
return nil
|
||||
}
|
||||
return ErrNoConfigMatch
|
||||
}
|
||||
|
||||
// RemoveQueueByArn removes all queue configurations that match the exact specified ARN
|
||||
func (b *Configuration) RemoveQueueByArn(arn Arn) {
|
||||
var queues []QueueConfig
|
||||
for _, queue := range b.QueueConfigs {
|
||||
if queue.Queue != arn.String() {
|
||||
queues = append(queues, queue)
|
||||
}
|
||||
}
|
||||
b.QueueConfigs = queues
|
||||
}
|
||||
|
||||
// RemoveQueueByArnEventsPrefixSuffix removes a queue configuration that match the exact specified ARN, events, prefix and suffix
|
||||
func (b *Configuration) RemoveQueueByArnEventsPrefixSuffix(arn Arn, events []EventType, prefix, suffix string) error {
|
||||
removeIndex := -1
|
||||
for i, v := range b.QueueConfigs {
|
||||
// if it matches events and filters, mark the index for deletion
|
||||
if v.Queue == arn.String() && v.Config.Equal(events, prefix, suffix) {
|
||||
removeIndex = i
|
||||
break // since we have at most one matching config
|
||||
}
|
||||
}
|
||||
if removeIndex >= 0 {
|
||||
b.QueueConfigs = append(b.QueueConfigs[:removeIndex], b.QueueConfigs[removeIndex+1:]...)
|
||||
return nil
|
||||
}
|
||||
return ErrNoConfigMatch
|
||||
}
|
||||
|
||||
// RemoveLambdaByArn removes all lambda configurations that match the exact specified ARN
|
||||
func (b *Configuration) RemoveLambdaByArn(arn Arn) {
|
||||
var lambdas []LambdaConfig
|
||||
for _, lambda := range b.LambdaConfigs {
|
||||
if lambda.Lambda != arn.String() {
|
||||
lambdas = append(lambdas, lambda)
|
||||
}
|
||||
}
|
||||
b.LambdaConfigs = lambdas
|
||||
}
|
||||
|
||||
// RemoveLambdaByArnEventsPrefixSuffix removes a topic configuration that match the exact specified ARN, events, prefix and suffix
|
||||
func (b *Configuration) RemoveLambdaByArnEventsPrefixSuffix(arn Arn, events []EventType, prefix, suffix string) error {
|
||||
removeIndex := -1
|
||||
for i, v := range b.LambdaConfigs {
|
||||
// if it matches events and filters, mark the index for deletion
|
||||
if v.Lambda == arn.String() && v.Config.Equal(events, prefix, suffix) {
|
||||
removeIndex = i
|
||||
break // since we have at most one matching config
|
||||
}
|
||||
}
|
||||
if removeIndex >= 0 {
|
||||
b.LambdaConfigs = append(b.LambdaConfigs[:removeIndex], b.LambdaConfigs[removeIndex+1:]...)
|
||||
return nil
|
||||
}
|
||||
return ErrNoConfigMatch
|
||||
}
|
||||
Generated
Vendored
+974
@@ -0,0 +1,974 @@
|
||||
/*
|
||||
* MinIO Client (C) 2020 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package replication
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/xml"
|
||||
"fmt"
|
||||
"math"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/rs/xid"
|
||||
)
|
||||
|
||||
var errInvalidFilter = fmt.Errorf("invalid filter")
|
||||
|
||||
// OptionType specifies operation to be performed on config
|
||||
type OptionType string
|
||||
|
||||
const (
|
||||
// AddOption specifies addition of rule to config
|
||||
AddOption OptionType = "Add"
|
||||
// SetOption specifies modification of existing rule to config
|
||||
SetOption OptionType = "Set"
|
||||
|
||||
// RemoveOption specifies rule options are for removing a rule
|
||||
RemoveOption OptionType = "Remove"
|
||||
// ImportOption is for getting current config
|
||||
ImportOption OptionType = "Import"
|
||||
)
|
||||
|
||||
// Options represents options to set a replication configuration rule
|
||||
type Options struct {
|
||||
Op OptionType
|
||||
RoleArn string
|
||||
ID string
|
||||
Prefix string
|
||||
RuleStatus string
|
||||
Priority string
|
||||
TagString string
|
||||
StorageClass string
|
||||
DestBucket string
|
||||
IsTagSet bool
|
||||
IsSCSet bool
|
||||
ReplicateDeletes string // replicate versioned deletes
|
||||
ReplicateDeleteMarkers string // replicate soft deletes
|
||||
ReplicaSync string // replicate replica metadata modifications
|
||||
ExistingObjectReplicate string
|
||||
}
|
||||
|
||||
// Tags returns a slice of tags for a rule
|
||||
func (opts Options) Tags() ([]Tag, error) {
|
||||
var tagList []Tag
|
||||
tagTokens := strings.Split(opts.TagString, "&")
|
||||
for _, tok := range tagTokens {
|
||||
if tok == "" {
|
||||
break
|
||||
}
|
||||
kv := strings.SplitN(tok, "=", 2)
|
||||
if len(kv) != 2 {
|
||||
return []Tag{}, fmt.Errorf("tags should be entered as comma separated k=v pairs")
|
||||
}
|
||||
tagList = append(tagList, Tag{
|
||||
Key: kv[0],
|
||||
Value: kv[1],
|
||||
})
|
||||
}
|
||||
return tagList, nil
|
||||
}
|
||||
|
||||
// Config - replication configuration specified in
|
||||
// https://docs.aws.amazon.com/AmazonS3/latest/dev/replication-add-config.html
|
||||
type Config struct {
|
||||
XMLName xml.Name `xml:"ReplicationConfiguration" json:"-"`
|
||||
Rules []Rule `xml:"Rule" json:"Rules"`
|
||||
Role string `xml:"Role" json:"Role"`
|
||||
}
|
||||
|
||||
// Empty returns true if config is not set
|
||||
func (c *Config) Empty() bool {
|
||||
return len(c.Rules) == 0
|
||||
}
|
||||
|
||||
// AddRule adds a new rule to existing replication config. If a rule exists with the
|
||||
// same ID, then the rule is replaced.
|
||||
func (c *Config) AddRule(opts Options) error {
|
||||
priority, err := strconv.Atoi(opts.Priority)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var compatSw bool // true if RoleArn is used with new mc client and older minio version prior to multisite
|
||||
if opts.RoleArn != "" {
|
||||
tokens := strings.Split(opts.RoleArn, ":")
|
||||
if len(tokens) != 6 {
|
||||
return fmt.Errorf("invalid format for replication Role Arn: %v", opts.RoleArn)
|
||||
}
|
||||
switch {
|
||||
case strings.HasPrefix(opts.RoleArn, "arn:minio:replication") && len(c.Rules) == 0:
|
||||
c.Role = opts.RoleArn
|
||||
compatSw = true
|
||||
case strings.HasPrefix(opts.RoleArn, "arn:aws:iam"):
|
||||
c.Role = opts.RoleArn
|
||||
default:
|
||||
return fmt.Errorf("RoleArn invalid for AWS replication configuration: %v", opts.RoleArn)
|
||||
}
|
||||
}
|
||||
|
||||
var status Status
|
||||
// toggle rule status for edit option
|
||||
switch opts.RuleStatus {
|
||||
case "enable":
|
||||
status = Enabled
|
||||
case "disable":
|
||||
status = Disabled
|
||||
default:
|
||||
return fmt.Errorf("rule state should be either [enable|disable]")
|
||||
}
|
||||
|
||||
tags, err := opts.Tags()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
andVal := And{
|
||||
Tags: tags,
|
||||
}
|
||||
filter := Filter{Prefix: opts.Prefix}
|
||||
// only a single tag is set.
|
||||
if opts.Prefix == "" && len(tags) == 1 {
|
||||
filter.Tag = tags[0]
|
||||
}
|
||||
// both prefix and tag are present
|
||||
if len(andVal.Tags) > 1 || opts.Prefix != "" {
|
||||
filter.And = andVal
|
||||
filter.And.Prefix = opts.Prefix
|
||||
filter.Prefix = ""
|
||||
filter.Tag = Tag{}
|
||||
}
|
||||
if opts.ID == "" {
|
||||
opts.ID = xid.New().String()
|
||||
}
|
||||
|
||||
destBucket := opts.DestBucket
|
||||
// ref https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-arn-format.html
|
||||
if btokens := strings.Split(destBucket, ":"); len(btokens) != 6 {
|
||||
if len(btokens) == 1 && compatSw {
|
||||
destBucket = fmt.Sprintf("arn:aws:s3:::%s", destBucket)
|
||||
} else {
|
||||
return fmt.Errorf("destination bucket needs to be in Arn format")
|
||||
}
|
||||
}
|
||||
dmStatus := Disabled
|
||||
if opts.ReplicateDeleteMarkers != "" {
|
||||
switch opts.ReplicateDeleteMarkers {
|
||||
case "enable":
|
||||
dmStatus = Enabled
|
||||
case "disable":
|
||||
dmStatus = Disabled
|
||||
default:
|
||||
return fmt.Errorf("ReplicateDeleteMarkers should be either enable|disable")
|
||||
}
|
||||
}
|
||||
|
||||
vDeleteStatus := Disabled
|
||||
if opts.ReplicateDeletes != "" {
|
||||
switch opts.ReplicateDeletes {
|
||||
case "enable":
|
||||
vDeleteStatus = Enabled
|
||||
case "disable":
|
||||
vDeleteStatus = Disabled
|
||||
default:
|
||||
return fmt.Errorf("ReplicateDeletes should be either enable|disable")
|
||||
}
|
||||
}
|
||||
var replicaSync Status
|
||||
// replica sync is by default Enabled, unless specified.
|
||||
switch opts.ReplicaSync {
|
||||
case "enable", "":
|
||||
replicaSync = Enabled
|
||||
case "disable":
|
||||
replicaSync = Disabled
|
||||
default:
|
||||
return fmt.Errorf("replica metadata sync should be either [enable|disable]")
|
||||
}
|
||||
|
||||
var existingStatus Status
|
||||
if opts.ExistingObjectReplicate != "" {
|
||||
switch opts.ExistingObjectReplicate {
|
||||
case "enable":
|
||||
existingStatus = Enabled
|
||||
case "disable", "":
|
||||
existingStatus = Disabled
|
||||
default:
|
||||
return fmt.Errorf("existingObjectReplicate should be either enable|disable")
|
||||
}
|
||||
}
|
||||
newRule := Rule{
|
||||
ID: opts.ID,
|
||||
Priority: priority,
|
||||
Status: status,
|
||||
Filter: filter,
|
||||
Destination: Destination{
|
||||
Bucket: destBucket,
|
||||
StorageClass: opts.StorageClass,
|
||||
},
|
||||
DeleteMarkerReplication: DeleteMarkerReplication{Status: dmStatus},
|
||||
DeleteReplication: DeleteReplication{Status: vDeleteStatus},
|
||||
// MinIO enables replica metadata syncing by default in the case of bi-directional replication to allow
|
||||
// automatic failover as the expectation in this case is that replica and source should be identical.
|
||||
// However AWS leaves this configurable https://docs.aws.amazon.com/AmazonS3/latest/dev/replication-for-metadata-changes.html
|
||||
SourceSelectionCriteria: SourceSelectionCriteria{
|
||||
ReplicaModifications: ReplicaModifications{
|
||||
Status: replicaSync,
|
||||
},
|
||||
},
|
||||
// By default disable existing object replication unless selected
|
||||
ExistingObjectReplication: ExistingObjectReplication{
|
||||
Status: existingStatus,
|
||||
},
|
||||
}
|
||||
|
||||
// validate rule after overlaying priority for pre-existing rule being disabled.
|
||||
if err := newRule.Validate(); err != nil {
|
||||
return err
|
||||
}
|
||||
// if replication config uses RoleArn, migrate this to the destination element as target ARN for remote bucket for MinIO configuration
|
||||
if c.Role != "" && !strings.HasPrefix(c.Role, "arn:aws:iam") && !compatSw {
|
||||
for i := range c.Rules {
|
||||
c.Rules[i].Destination.Bucket = c.Role
|
||||
}
|
||||
c.Role = ""
|
||||
}
|
||||
|
||||
for _, rule := range c.Rules {
|
||||
if rule.Priority == newRule.Priority {
|
||||
return fmt.Errorf("priority must be unique. Replication configuration already has a rule with this priority")
|
||||
}
|
||||
if rule.ID == newRule.ID {
|
||||
return fmt.Errorf("a rule exists with this ID")
|
||||
}
|
||||
}
|
||||
|
||||
c.Rules = append(c.Rules, newRule)
|
||||
return nil
|
||||
}
|
||||
|
||||
// EditRule modifies an existing rule in replication config
|
||||
func (c *Config) EditRule(opts Options) error {
|
||||
if opts.ID == "" {
|
||||
return fmt.Errorf("rule ID missing")
|
||||
}
|
||||
// if replication config uses RoleArn, migrate this to the destination element as target ARN for remote bucket for non AWS.
|
||||
if c.Role != "" && !strings.HasPrefix(c.Role, "arn:aws:iam") && len(c.Rules) > 1 {
|
||||
for i := range c.Rules {
|
||||
c.Rules[i].Destination.Bucket = c.Role
|
||||
}
|
||||
c.Role = ""
|
||||
}
|
||||
|
||||
rIdx := -1
|
||||
var newRule Rule
|
||||
for i, rule := range c.Rules {
|
||||
if rule.ID == opts.ID {
|
||||
rIdx = i
|
||||
newRule = rule
|
||||
break
|
||||
}
|
||||
}
|
||||
if rIdx < 0 {
|
||||
return fmt.Errorf("rule with ID %s not found in replication configuration", opts.ID)
|
||||
}
|
||||
prefixChg := opts.Prefix != newRule.Prefix()
|
||||
if opts.IsTagSet || prefixChg {
|
||||
prefix := newRule.Prefix()
|
||||
if prefix != opts.Prefix {
|
||||
prefix = opts.Prefix
|
||||
}
|
||||
tags := []Tag{newRule.Filter.Tag}
|
||||
if len(newRule.Filter.And.Tags) != 0 {
|
||||
tags = newRule.Filter.And.Tags
|
||||
}
|
||||
var err error
|
||||
if opts.IsTagSet {
|
||||
tags, err = opts.Tags()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
andVal := And{
|
||||
Tags: tags,
|
||||
}
|
||||
|
||||
filter := Filter{Prefix: prefix}
|
||||
// only a single tag is set.
|
||||
if prefix == "" && len(tags) == 1 {
|
||||
filter.Tag = tags[0]
|
||||
}
|
||||
// both prefix and tag are present
|
||||
if len(andVal.Tags) > 1 || prefix != "" {
|
||||
filter.And = andVal
|
||||
filter.And.Prefix = prefix
|
||||
filter.Prefix = ""
|
||||
filter.Tag = Tag{}
|
||||
}
|
||||
newRule.Filter = filter
|
||||
}
|
||||
|
||||
// toggle rule status for edit option
|
||||
if opts.RuleStatus != "" {
|
||||
switch opts.RuleStatus {
|
||||
case "enable":
|
||||
newRule.Status = Enabled
|
||||
case "disable":
|
||||
newRule.Status = Disabled
|
||||
default:
|
||||
return fmt.Errorf("rule state should be either [enable|disable]")
|
||||
}
|
||||
}
|
||||
// set DeleteMarkerReplication rule status for edit option
|
||||
if opts.ReplicateDeleteMarkers != "" {
|
||||
switch opts.ReplicateDeleteMarkers {
|
||||
case "enable":
|
||||
newRule.DeleteMarkerReplication.Status = Enabled
|
||||
case "disable":
|
||||
newRule.DeleteMarkerReplication.Status = Disabled
|
||||
default:
|
||||
return fmt.Errorf("ReplicateDeleteMarkers state should be either [enable|disable]")
|
||||
}
|
||||
}
|
||||
|
||||
// set DeleteReplication rule status for edit option. This is a MinIO specific
|
||||
// option to replicate versioned deletes
|
||||
if opts.ReplicateDeletes != "" {
|
||||
switch opts.ReplicateDeletes {
|
||||
case "enable":
|
||||
newRule.DeleteReplication.Status = Enabled
|
||||
case "disable":
|
||||
newRule.DeleteReplication.Status = Disabled
|
||||
default:
|
||||
return fmt.Errorf("ReplicateDeletes state should be either [enable|disable]")
|
||||
}
|
||||
}
|
||||
|
||||
if opts.ReplicaSync != "" {
|
||||
switch opts.ReplicaSync {
|
||||
case "enable", "":
|
||||
newRule.SourceSelectionCriteria.ReplicaModifications.Status = Enabled
|
||||
case "disable":
|
||||
newRule.SourceSelectionCriteria.ReplicaModifications.Status = Disabled
|
||||
default:
|
||||
return fmt.Errorf("replica metadata sync should be either [enable|disable]")
|
||||
}
|
||||
}
|
||||
|
||||
if opts.ExistingObjectReplicate != "" {
|
||||
switch opts.ExistingObjectReplicate {
|
||||
case "enable":
|
||||
newRule.ExistingObjectReplication.Status = Enabled
|
||||
case "disable":
|
||||
newRule.ExistingObjectReplication.Status = Disabled
|
||||
default:
|
||||
return fmt.Errorf("existingObjectsReplication state should be either [enable|disable]")
|
||||
}
|
||||
}
|
||||
if opts.IsSCSet {
|
||||
newRule.Destination.StorageClass = opts.StorageClass
|
||||
}
|
||||
if opts.Priority != "" {
|
||||
priority, err := strconv.Atoi(opts.Priority)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
newRule.Priority = priority
|
||||
}
|
||||
if opts.DestBucket != "" {
|
||||
destBucket := opts.DestBucket
|
||||
// ref https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-arn-format.html
|
||||
if btokens := strings.Split(opts.DestBucket, ":"); len(btokens) != 6 {
|
||||
return fmt.Errorf("destination bucket needs to be in Arn format")
|
||||
}
|
||||
newRule.Destination.Bucket = destBucket
|
||||
}
|
||||
// validate rule
|
||||
if err := newRule.Validate(); err != nil {
|
||||
return err
|
||||
}
|
||||
// ensure priority and destination bucket restrictions are not violated
|
||||
for idx, rule := range c.Rules {
|
||||
if rule.Priority == newRule.Priority && rIdx != idx {
|
||||
return fmt.Errorf("priority must be unique. Replication configuration already has a rule with this priority")
|
||||
}
|
||||
if rule.Destination.Bucket != newRule.Destination.Bucket && rule.ID == newRule.ID {
|
||||
if c.Role == newRule.Destination.Bucket {
|
||||
continue
|
||||
}
|
||||
return fmt.Errorf("invalid destination bucket for this rule")
|
||||
}
|
||||
}
|
||||
|
||||
c.Rules[rIdx] = newRule
|
||||
return nil
|
||||
}
|
||||
|
||||
// RemoveRule removes a rule from replication config.
|
||||
func (c *Config) RemoveRule(opts Options) error {
|
||||
var newRules []Rule
|
||||
ruleFound := false
|
||||
for _, rule := range c.Rules {
|
||||
if rule.ID != opts.ID {
|
||||
newRules = append(newRules, rule)
|
||||
continue
|
||||
}
|
||||
ruleFound = true
|
||||
}
|
||||
if !ruleFound {
|
||||
return fmt.Errorf("Rule with ID %s not found", opts.ID)
|
||||
}
|
||||
if len(newRules) == 0 {
|
||||
return fmt.Errorf("replication configuration should have at least one rule")
|
||||
}
|
||||
c.Rules = newRules
|
||||
return nil
|
||||
}
|
||||
|
||||
// Rule - a rule for replication configuration.
|
||||
type Rule struct {
|
||||
XMLName xml.Name `xml:"Rule" json:"-"`
|
||||
ID string `xml:"ID,omitempty"`
|
||||
Status Status `xml:"Status"`
|
||||
Priority int `xml:"Priority"`
|
||||
DeleteMarkerReplication DeleteMarkerReplication `xml:"DeleteMarkerReplication"`
|
||||
DeleteReplication DeleteReplication `xml:"DeleteReplication"`
|
||||
Destination Destination `xml:"Destination"`
|
||||
Filter Filter `xml:"Filter" json:"Filter"`
|
||||
SourceSelectionCriteria SourceSelectionCriteria `xml:"SourceSelectionCriteria" json:"SourceSelectionCriteria"`
|
||||
ExistingObjectReplication ExistingObjectReplication `xml:"ExistingObjectReplication,omitempty" json:"ExistingObjectReplication,omitempty"`
|
||||
}
|
||||
|
||||
// Validate validates the rule for correctness
|
||||
func (r Rule) Validate() error {
|
||||
if err := r.validateID(); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := r.validateStatus(); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := r.validateFilter(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if r.Priority < 0 && r.Status == Enabled {
|
||||
return fmt.Errorf("priority must be set for the rule")
|
||||
}
|
||||
|
||||
if err := r.validateStatus(); err != nil {
|
||||
return err
|
||||
}
|
||||
return r.ExistingObjectReplication.Validate()
|
||||
}
|
||||
|
||||
// validateID - checks if ID is valid or not.
|
||||
func (r Rule) validateID() error {
|
||||
// cannot be longer than 255 characters
|
||||
if len(r.ID) > 255 {
|
||||
return fmt.Errorf("ID must be less than 255 characters")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// validateStatus - checks if status is valid or not.
|
||||
func (r Rule) validateStatus() error {
|
||||
// Status can't be empty
|
||||
if len(r.Status) == 0 {
|
||||
return fmt.Errorf("status cannot be empty")
|
||||
}
|
||||
|
||||
// Status must be one of Enabled or Disabled
|
||||
if r.Status != Enabled && r.Status != Disabled {
|
||||
return fmt.Errorf("status must be set to either Enabled or Disabled")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (r Rule) validateFilter() error {
|
||||
return r.Filter.Validate()
|
||||
}
|
||||
|
||||
// Prefix - a rule can either have prefix under <filter></filter> or under
|
||||
// <filter><and></and></filter>. This method returns the prefix from the
|
||||
// location where it is available
|
||||
func (r Rule) Prefix() string {
|
||||
if r.Filter.Prefix != "" {
|
||||
return r.Filter.Prefix
|
||||
}
|
||||
return r.Filter.And.Prefix
|
||||
}
|
||||
|
||||
// Tags - a rule can either have tag under <filter></filter> or under
|
||||
// <filter><and></and></filter>. This method returns all the tags from the
|
||||
// rule in the format tag1=value1&tag2=value2
|
||||
func (r Rule) Tags() string {
|
||||
ts := []Tag{r.Filter.Tag}
|
||||
if len(r.Filter.And.Tags) != 0 {
|
||||
ts = r.Filter.And.Tags
|
||||
}
|
||||
|
||||
var buf bytes.Buffer
|
||||
for _, t := range ts {
|
||||
if buf.Len() > 0 {
|
||||
buf.WriteString("&")
|
||||
}
|
||||
buf.WriteString(t.String())
|
||||
}
|
||||
return buf.String()
|
||||
}
|
||||
|
||||
// Filter - a filter for a replication configuration Rule.
|
||||
type Filter struct {
|
||||
XMLName xml.Name `xml:"Filter" json:"-"`
|
||||
Prefix string `json:"Prefix,omitempty"`
|
||||
And And `xml:"And,omitempty" json:"And,omitempty"`
|
||||
Tag Tag `xml:"Tag,omitempty" json:"Tag,omitempty"`
|
||||
}
|
||||
|
||||
// Validate - validates the filter element
|
||||
func (f Filter) Validate() error {
|
||||
// A Filter must have exactly one of Prefix, Tag, or And specified.
|
||||
if !f.And.isEmpty() {
|
||||
if f.Prefix != "" {
|
||||
return errInvalidFilter
|
||||
}
|
||||
if !f.Tag.IsEmpty() {
|
||||
return errInvalidFilter
|
||||
}
|
||||
}
|
||||
if f.Prefix != "" {
|
||||
if !f.Tag.IsEmpty() {
|
||||
return errInvalidFilter
|
||||
}
|
||||
}
|
||||
if !f.Tag.IsEmpty() {
|
||||
if err := f.Tag.Validate(); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Tag - a tag for a replication configuration Rule filter.
|
||||
type Tag struct {
|
||||
XMLName xml.Name `json:"-"`
|
||||
Key string `xml:"Key,omitempty" json:"Key,omitempty"`
|
||||
Value string `xml:"Value,omitempty" json:"Value,omitempty"`
|
||||
}
|
||||
|
||||
func (tag Tag) String() string {
|
||||
if tag.IsEmpty() {
|
||||
return ""
|
||||
}
|
||||
return tag.Key + "=" + tag.Value
|
||||
}
|
||||
|
||||
// IsEmpty returns whether this tag is empty or not.
|
||||
func (tag Tag) IsEmpty() bool {
|
||||
return tag.Key == ""
|
||||
}
|
||||
|
||||
// Validate checks this tag.
|
||||
func (tag Tag) Validate() error {
|
||||
if len(tag.Key) == 0 || utf8.RuneCountInString(tag.Key) > 128 {
|
||||
return fmt.Errorf("invalid Tag Key")
|
||||
}
|
||||
|
||||
if utf8.RuneCountInString(tag.Value) > 256 {
|
||||
return fmt.Errorf("invalid Tag Value")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Destination - destination in ReplicationConfiguration.
|
||||
type Destination struct {
|
||||
XMLName xml.Name `xml:"Destination" json:"-"`
|
||||
Bucket string `xml:"Bucket" json:"Bucket"`
|
||||
StorageClass string `xml:"StorageClass,omitempty" json:"StorageClass,omitempty"`
|
||||
}
|
||||
|
||||
// And - a tag to combine a prefix and multiple tags for replication configuration rule.
|
||||
type And struct {
|
||||
XMLName xml.Name `xml:"And,omitempty" json:"-"`
|
||||
Prefix string `xml:"Prefix,omitempty" json:"Prefix,omitempty"`
|
||||
Tags []Tag `xml:"Tag,omitempty" json:"Tag,omitempty"`
|
||||
}
|
||||
|
||||
// isEmpty returns true if Tags field is null
|
||||
func (a And) isEmpty() bool {
|
||||
return len(a.Tags) == 0 && a.Prefix == ""
|
||||
}
|
||||
|
||||
// Status represents Enabled/Disabled status
|
||||
type Status string
|
||||
|
||||
// Supported status types
|
||||
const (
|
||||
Enabled Status = "Enabled"
|
||||
Disabled Status = "Disabled"
|
||||
)
|
||||
|
||||
// DeleteMarkerReplication - whether delete markers are replicated - https://docs.aws.amazon.com/AmazonS3/latest/dev/replication-add-config.html
|
||||
type DeleteMarkerReplication struct {
|
||||
Status Status `xml:"Status" json:"Status"` // should be set to "Disabled" by default
|
||||
}
|
||||
|
||||
// IsEmpty returns true if DeleteMarkerReplication is not set
|
||||
func (d DeleteMarkerReplication) IsEmpty() bool {
|
||||
return len(d.Status) == 0
|
||||
}
|
||||
|
||||
// DeleteReplication - whether versioned deletes are replicated - this
|
||||
// is a MinIO specific extension
|
||||
type DeleteReplication struct {
|
||||
Status Status `xml:"Status" json:"Status"` // should be set to "Disabled" by default
|
||||
}
|
||||
|
||||
// IsEmpty returns true if DeleteReplication is not set
|
||||
func (d DeleteReplication) IsEmpty() bool {
|
||||
return len(d.Status) == 0
|
||||
}
|
||||
|
||||
// ReplicaModifications specifies if replica modification sync is enabled
|
||||
type ReplicaModifications struct {
|
||||
Status Status `xml:"Status" json:"Status"` // should be set to "Enabled" by default
|
||||
}
|
||||
|
||||
// SourceSelectionCriteria - specifies additional source selection criteria in ReplicationConfiguration.
|
||||
type SourceSelectionCriteria struct {
|
||||
ReplicaModifications ReplicaModifications `xml:"ReplicaModifications" json:"ReplicaModifications"`
|
||||
}
|
||||
|
||||
// IsValid - checks whether SourceSelectionCriteria is valid or not.
|
||||
func (s SourceSelectionCriteria) IsValid() bool {
|
||||
return s.ReplicaModifications.Status == Enabled || s.ReplicaModifications.Status == Disabled
|
||||
}
|
||||
|
||||
// Validate source selection criteria
|
||||
func (s SourceSelectionCriteria) Validate() error {
|
||||
if (s == SourceSelectionCriteria{}) {
|
||||
return nil
|
||||
}
|
||||
if !s.IsValid() {
|
||||
return fmt.Errorf("invalid ReplicaModification status")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ExistingObjectReplication - whether existing object replication is enabled
|
||||
type ExistingObjectReplication struct {
|
||||
Status Status `xml:"Status"` // should be set to "Disabled" by default
|
||||
}
|
||||
|
||||
// IsEmpty returns true if DeleteMarkerReplication is not set
|
||||
func (e ExistingObjectReplication) IsEmpty() bool {
|
||||
return len(e.Status) == 0
|
||||
}
|
||||
|
||||
// Validate validates whether the status is disabled.
|
||||
func (e ExistingObjectReplication) Validate() error {
|
||||
if e.IsEmpty() {
|
||||
return nil
|
||||
}
|
||||
if e.Status != Disabled && e.Status != Enabled {
|
||||
return fmt.Errorf("invalid ExistingObjectReplication status")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// TargetMetrics represents inline replication metrics
|
||||
// such as pending, failed and completed bytes in total for a bucket remote target
|
||||
type TargetMetrics struct {
|
||||
// Completed count
|
||||
ReplicatedCount uint64 `json:"replicationCount,omitempty"`
|
||||
// Completed size in bytes
|
||||
ReplicatedSize uint64 `json:"completedReplicationSize,omitempty"`
|
||||
// Bandwidth limit in bytes/sec for this target
|
||||
BandWidthLimitInBytesPerSecond int64 `json:"limitInBits,omitempty"`
|
||||
// Current bandwidth used in bytes/sec for this target
|
||||
CurrentBandwidthInBytesPerSecond float64 `json:"currentBandwidth,omitempty"`
|
||||
// errors seen in replication in last minute, hour and total
|
||||
Failed TimedErrStats `json:"failed,omitempty"`
|
||||
// Deprecated fields
|
||||
// Pending size in bytes
|
||||
PendingSize uint64 `json:"pendingReplicationSize,omitempty"`
|
||||
// Total Replica size in bytes
|
||||
ReplicaSize uint64 `json:"replicaSize,omitempty"`
|
||||
// Failed size in bytes
|
||||
FailedSize uint64 `json:"failedReplicationSize,omitempty"`
|
||||
// Total number of pending operations including metadata updates
|
||||
PendingCount uint64 `json:"pendingReplicationCount,omitempty"`
|
||||
// Total number of failed operations including metadata updates
|
||||
FailedCount uint64 `json:"failedReplicationCount,omitempty"`
|
||||
}
|
||||
|
||||
// Metrics represents inline replication metrics for a bucket.
|
||||
type Metrics struct {
|
||||
Stats map[string]TargetMetrics
|
||||
// Completed size in bytes across targets
|
||||
ReplicatedSize uint64 `json:"completedReplicationSize,omitempty"`
|
||||
// Total Replica size in bytes across targets
|
||||
ReplicaSize uint64 `json:"replicaSize,omitempty"`
|
||||
// Total Replica counts
|
||||
ReplicaCount int64 `json:"replicaCount,omitempty"`
|
||||
// Total Replicated count
|
||||
ReplicatedCount int64 `json:"replicationCount,omitempty"`
|
||||
// errors seen in replication in last minute, hour and total
|
||||
Errors TimedErrStats `json:"failed,omitempty"`
|
||||
// Total number of entries that are queued for replication
|
||||
QStats InQueueMetric `json:"queued"`
|
||||
// Deprecated fields
|
||||
// Total Pending size in bytes across targets
|
||||
PendingSize uint64 `json:"pendingReplicationSize,omitempty"`
|
||||
// Failed size in bytes across targets
|
||||
FailedSize uint64 `json:"failedReplicationSize,omitempty"`
|
||||
// Total number of pending operations including metadata updates across targets
|
||||
PendingCount uint64 `json:"pendingReplicationCount,omitempty"`
|
||||
// Total number of failed operations including metadata updates across targets
|
||||
FailedCount uint64 `json:"failedReplicationCount,omitempty"`
|
||||
}
|
||||
|
||||
// RStat - has count and bytes for replication metrics
|
||||
type RStat struct {
|
||||
Count float64 `json:"count"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
}
|
||||
|
||||
// Add two RStat
|
||||
func (r RStat) Add(r1 RStat) RStat {
|
||||
return RStat{
|
||||
Count: r.Count + r1.Count,
|
||||
Bytes: r.Bytes + r1.Bytes,
|
||||
}
|
||||
}
|
||||
|
||||
// TimedErrStats holds error stats for a time period
|
||||
type TimedErrStats struct {
|
||||
LastMinute RStat `json:"lastMinute"`
|
||||
LastHour RStat `json:"lastHour"`
|
||||
Totals RStat `json:"totals"`
|
||||
}
|
||||
|
||||
// Add two TimedErrStats
|
||||
func (te TimedErrStats) Add(o TimedErrStats) TimedErrStats {
|
||||
return TimedErrStats{
|
||||
LastMinute: te.LastMinute.Add(o.LastMinute),
|
||||
LastHour: te.LastHour.Add(o.LastHour),
|
||||
Totals: te.Totals.Add(o.Totals),
|
||||
}
|
||||
}
|
||||
|
||||
// ResyncTargetsInfo provides replication target information to resync replicated data.
|
||||
type ResyncTargetsInfo struct {
|
||||
Targets []ResyncTarget `json:"target,omitempty"`
|
||||
}
|
||||
|
||||
// ResyncTarget provides the replica resources and resetID to initiate resync replication.
|
||||
type ResyncTarget struct {
|
||||
Arn string `json:"arn"`
|
||||
ResetID string `json:"resetid"`
|
||||
StartTime time.Time `json:"startTime,omitempty"`
|
||||
EndTime time.Time `json:"endTime,omitempty"`
|
||||
// Status of resync operation
|
||||
ResyncStatus string `json:"resyncStatus,omitempty"`
|
||||
// Completed size in bytes
|
||||
ReplicatedSize int64 `json:"completedReplicationSize,omitempty"`
|
||||
// Failed size in bytes
|
||||
FailedSize int64 `json:"failedReplicationSize,omitempty"`
|
||||
// Total number of failed operations
|
||||
FailedCount int64 `json:"failedReplicationCount,omitempty"`
|
||||
// Total number of completed operations
|
||||
ReplicatedCount int64 `json:"replicationCount,omitempty"`
|
||||
// Last bucket/object replicated.
|
||||
Bucket string `json:"bucket,omitempty"`
|
||||
Object string `json:"object,omitempty"`
|
||||
}
|
||||
|
||||
// XferStats holds transfer rate info for uploads/sec
|
||||
type XferStats struct {
|
||||
AvgRate float64 `json:"avgRate"`
|
||||
PeakRate float64 `json:"peakRate"`
|
||||
CurrRate float64 `json:"currRate"`
|
||||
}
|
||||
|
||||
// Merge two XferStats
|
||||
func (x *XferStats) Merge(x1 XferStats) {
|
||||
x.AvgRate += x1.AvgRate
|
||||
x.PeakRate += x1.PeakRate
|
||||
x.CurrRate += x1.CurrRate
|
||||
}
|
||||
|
||||
// QStat holds count and bytes for objects in replication queue
|
||||
type QStat struct {
|
||||
Count float64 `json:"count"`
|
||||
Bytes float64 `json:"bytes"`
|
||||
}
|
||||
|
||||
// Add 2 QStat entries
|
||||
func (q *QStat) Add(q1 QStat) {
|
||||
q.Count += q1.Count
|
||||
q.Bytes += q1.Bytes
|
||||
}
|
||||
|
||||
// InQueueMetric holds stats for objects in replication queue
|
||||
type InQueueMetric struct {
|
||||
Curr QStat `json:"curr" msg:"cq"`
|
||||
Avg QStat `json:"avg" msg:"aq"`
|
||||
Max QStat `json:"peak" msg:"pq"`
|
||||
}
|
||||
|
||||
// MetricName name of replication metric
|
||||
type MetricName string
|
||||
|
||||
const (
|
||||
// Large is a metric name for large objects >=128MiB
|
||||
Large MetricName = "Large"
|
||||
// Small is a metric name for objects <128MiB size
|
||||
Small MetricName = "Small"
|
||||
// Total is a metric name for total objects
|
||||
Total MetricName = "Total"
|
||||
)
|
||||
|
||||
// WorkerStat has stats on number of replication workers
|
||||
type WorkerStat struct {
|
||||
Curr int32 `json:"curr"`
|
||||
Avg float32 `json:"avg"`
|
||||
Max int32 `json:"max"`
|
||||
}
|
||||
|
||||
// ReplMRFStats holds stats of MRF backlog saved to disk in the last 5 minutes
|
||||
// and number of entries that failed replication after 3 retries
|
||||
type ReplMRFStats struct {
|
||||
LastFailedCount uint64 `json:"failedCount_last5min"`
|
||||
// Count of unreplicated entries that were dropped after MRF retry limit reached since cluster start.
|
||||
TotalDroppedCount uint64 `json:"droppedCount_since_uptime"`
|
||||
// Bytes of unreplicated entries that were dropped after MRF retry limit reached since cluster start.
|
||||
TotalDroppedBytes uint64 `json:"droppedBytes_since_uptime"`
|
||||
}
|
||||
|
||||
// ReplQNodeStats holds stats for a node in replication queue
|
||||
type ReplQNodeStats struct {
|
||||
NodeName string `json:"nodeName"`
|
||||
Uptime int64 `json:"uptime"`
|
||||
Workers WorkerStat `json:"activeWorkers"`
|
||||
|
||||
XferStats map[MetricName]XferStats `json:"transferSummary"`
|
||||
TgtXferStats map[string]map[MetricName]XferStats `json:"tgtTransferStats"`
|
||||
|
||||
QStats InQueueMetric `json:"queueStats"`
|
||||
MRFStats ReplMRFStats `json:"mrfStats"`
|
||||
}
|
||||
|
||||
// ReplQueueStats holds stats for replication queue across nodes
|
||||
type ReplQueueStats struct {
|
||||
Nodes []ReplQNodeStats `json:"nodes"`
|
||||
}
|
||||
|
||||
// Workers returns number of workers across all nodes
|
||||
func (q ReplQueueStats) Workers() (tot WorkerStat) {
|
||||
for _, node := range q.Nodes {
|
||||
tot.Avg += node.Workers.Avg
|
||||
tot.Curr += node.Workers.Curr
|
||||
if tot.Max < node.Workers.Max {
|
||||
tot.Max = node.Workers.Max
|
||||
}
|
||||
}
|
||||
if len(q.Nodes) > 0 {
|
||||
tot.Avg /= float32(len(q.Nodes))
|
||||
tot.Curr /= int32(len(q.Nodes))
|
||||
}
|
||||
return tot
|
||||
}
|
||||
|
||||
// qStatSummary returns cluster level stats for objects in replication queue
|
||||
func (q ReplQueueStats) qStatSummary() InQueueMetric {
|
||||
m := InQueueMetric{}
|
||||
for _, v := range q.Nodes {
|
||||
m.Avg.Add(v.QStats.Avg)
|
||||
m.Curr.Add(v.QStats.Curr)
|
||||
if m.Max.Count < v.QStats.Max.Count {
|
||||
m.Max.Add(v.QStats.Max)
|
||||
}
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// ReplQStats holds stats for objects in replication queue
|
||||
type ReplQStats struct {
|
||||
Uptime int64 `json:"uptime"`
|
||||
Workers WorkerStat `json:"workers"`
|
||||
|
||||
XferStats map[MetricName]XferStats `json:"xferStats"`
|
||||
TgtXferStats map[string]map[MetricName]XferStats `json:"tgtXferStats"`
|
||||
|
||||
QStats InQueueMetric `json:"qStats"`
|
||||
MRFStats ReplMRFStats `json:"mrfStats"`
|
||||
}
|
||||
|
||||
// QStats returns cluster level stats for objects in replication queue
|
||||
func (q ReplQueueStats) QStats() (r ReplQStats) {
|
||||
r.QStats = q.qStatSummary()
|
||||
r.XferStats = make(map[MetricName]XferStats)
|
||||
r.TgtXferStats = make(map[string]map[MetricName]XferStats)
|
||||
r.Workers = q.Workers()
|
||||
|
||||
for _, node := range q.Nodes {
|
||||
for arn := range node.TgtXferStats {
|
||||
xmap, ok := node.TgtXferStats[arn]
|
||||
if !ok {
|
||||
xmap = make(map[MetricName]XferStats)
|
||||
}
|
||||
for m, v := range xmap {
|
||||
st, ok := r.XferStats[m]
|
||||
if !ok {
|
||||
st = XferStats{}
|
||||
}
|
||||
st.AvgRate += v.AvgRate
|
||||
st.CurrRate += v.CurrRate
|
||||
st.PeakRate = math.Max(st.PeakRate, v.PeakRate)
|
||||
if _, ok := r.TgtXferStats[arn]; !ok {
|
||||
r.TgtXferStats[arn] = make(map[MetricName]XferStats)
|
||||
}
|
||||
r.TgtXferStats[arn][m] = st
|
||||
}
|
||||
}
|
||||
for k, v := range node.XferStats {
|
||||
st, ok := r.XferStats[k]
|
||||
if !ok {
|
||||
st = XferStats{}
|
||||
}
|
||||
st.AvgRate += v.AvgRate
|
||||
st.CurrRate += v.CurrRate
|
||||
st.PeakRate = math.Max(st.PeakRate, v.PeakRate)
|
||||
r.XferStats[k] = st
|
||||
}
|
||||
r.MRFStats.LastFailedCount += node.MRFStats.LastFailedCount
|
||||
r.MRFStats.TotalDroppedCount += node.MRFStats.TotalDroppedCount
|
||||
r.MRFStats.TotalDroppedBytes += node.MRFStats.TotalDroppedBytes
|
||||
r.Uptime += node.Uptime
|
||||
}
|
||||
if len(q.Nodes) > 0 {
|
||||
r.Uptime /= int64(len(q.Nodes)) // average uptime
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// MetricsV2 represents replication metrics for a bucket.
|
||||
type MetricsV2 struct {
|
||||
Uptime int64 `json:"uptime"`
|
||||
CurrentStats Metrics `json:"currStats"`
|
||||
QueueStats ReplQueueStats `json:"queueStats"`
|
||||
}
|
||||
+411
@@ -0,0 +1,411 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2015-2020 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package s3utils
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"net"
|
||||
"net/url"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
// Sentinel URL is the default url value which is invalid.
|
||||
var sentinelURL = url.URL{}
|
||||
|
||||
// IsValidDomain validates if input string is a valid domain name.
|
||||
func IsValidDomain(host string) bool {
|
||||
// See RFC 1035, RFC 3696.
|
||||
host = strings.TrimSpace(host)
|
||||
if len(host) == 0 || len(host) > 255 {
|
||||
return false
|
||||
}
|
||||
// host cannot start or end with "-"
|
||||
if host[len(host)-1:] == "-" || host[:1] == "-" {
|
||||
return false
|
||||
}
|
||||
// host cannot start or end with "_"
|
||||
if host[len(host)-1:] == "_" || host[:1] == "_" {
|
||||
return false
|
||||
}
|
||||
// host cannot start with a "."
|
||||
if host[:1] == "." {
|
||||
return false
|
||||
}
|
||||
// All non alphanumeric characters are invalid.
|
||||
if strings.ContainsAny(host, "`~!@#$%^&*()+={}[]|\\\"';:><?/") {
|
||||
return false
|
||||
}
|
||||
// No need to regexp match, since the list is non-exhaustive.
|
||||
// We let it valid and fail later.
|
||||
return true
|
||||
}
|
||||
|
||||
// IsValidIP parses input string for ip address validity.
|
||||
func IsValidIP(ip string) bool {
|
||||
return net.ParseIP(ip) != nil
|
||||
}
|
||||
|
||||
// IsVirtualHostSupported - verifies if bucketName can be part of
|
||||
// virtual host. Currently only Amazon S3 and Google Cloud Storage
|
||||
// would support this.
|
||||
func IsVirtualHostSupported(endpointURL url.URL, bucketName string) bool {
|
||||
if endpointURL == sentinelURL {
|
||||
return false
|
||||
}
|
||||
// bucketName can be valid but '.' in the hostname will fail SSL
|
||||
// certificate validation. So do not use host-style for such buckets.
|
||||
if endpointURL.Scheme == "https" && strings.Contains(bucketName, ".") {
|
||||
return false
|
||||
}
|
||||
// Return true for all other cases
|
||||
return IsAmazonEndpoint(endpointURL) || IsGoogleEndpoint(endpointURL) || IsAliyunOSSEndpoint(endpointURL)
|
||||
}
|
||||
|
||||
// Refer for region styles - https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
|
||||
|
||||
// amazonS3HostHyphen - regular expression used to determine if an arg is s3 host in hyphenated style.
|
||||
var amazonS3HostHyphen = regexp.MustCompile(`^s3-(.*?).amazonaws.com$`)
|
||||
|
||||
// amazonS3HostDualStack - regular expression used to determine if an arg is s3 host dualstack.
|
||||
var amazonS3HostDualStack = regexp.MustCompile(`^s3.dualstack.(.*?).amazonaws.com$`)
|
||||
|
||||
// amazonS3HostFIPS - regular expression used to determine if an arg is s3 FIPS host.
|
||||
var amazonS3HostFIPS = regexp.MustCompile(`^s3-fips.(.*?).amazonaws.com$`)
|
||||
|
||||
// amazonS3HostFIPSDualStack - regular expression used to determine if an arg is s3 FIPS host dualstack.
|
||||
var amazonS3HostFIPSDualStack = regexp.MustCompile(`^s3-fips.dualstack.(.*?).amazonaws.com$`)
|
||||
|
||||
// amazonS3HostDot - regular expression used to determine if an arg is s3 host in . style.
|
||||
var amazonS3HostDot = regexp.MustCompile(`^s3.(.*?).amazonaws.com$`)
|
||||
|
||||
// amazonS3ChinaHost - regular expression used to determine if the arg is s3 china host.
|
||||
var amazonS3ChinaHost = regexp.MustCompile(`^s3.(cn.*?).amazonaws.com.cn$`)
|
||||
|
||||
// amazonS3ChinaHostDualStack - regular expression used to determine if the arg is s3 china host dualstack.
|
||||
var amazonS3ChinaHostDualStack = regexp.MustCompile(`^s3.dualstack.(cn.*?).amazonaws.com.cn$`)
|
||||
|
||||
// Regular expression used to determine if the arg is elb host.
|
||||
var elbAmazonRegex = regexp.MustCompile(`elb(.*?).amazonaws.com$`)
|
||||
|
||||
// Regular expression used to determine if the arg is elb host in china.
|
||||
var elbAmazonCnRegex = regexp.MustCompile(`elb(.*?).amazonaws.com.cn$`)
|
||||
|
||||
// amazonS3HostPrivateLink - regular expression used to determine if an arg is s3 host in AWS PrivateLink interface endpoints style
|
||||
var amazonS3HostPrivateLink = regexp.MustCompile(`^(?:bucket|accesspoint).vpce-.*?.s3.(.*?).vpce.amazonaws.com$`)
|
||||
|
||||
// GetRegionFromURL - returns a region from url host.
|
||||
func GetRegionFromURL(endpointURL url.URL) string {
|
||||
if endpointURL == sentinelURL {
|
||||
return ""
|
||||
}
|
||||
if endpointURL.Host == "s3-external-1.amazonaws.com" {
|
||||
return ""
|
||||
}
|
||||
|
||||
// if elb's are used we cannot calculate which region it may be, just return empty.
|
||||
if elbAmazonRegex.MatchString(endpointURL.Host) || elbAmazonCnRegex.MatchString(endpointURL.Host) {
|
||||
return ""
|
||||
}
|
||||
|
||||
// We check for FIPS dualstack matching first to avoid the non-greedy
|
||||
// regex for FIPS non-dualstack matching a dualstack URL
|
||||
parts := amazonS3HostFIPSDualStack.FindStringSubmatch(endpointURL.Host)
|
||||
if len(parts) > 1 {
|
||||
return parts[1]
|
||||
}
|
||||
|
||||
parts = amazonS3HostFIPS.FindStringSubmatch(endpointURL.Host)
|
||||
if len(parts) > 1 {
|
||||
return parts[1]
|
||||
}
|
||||
|
||||
parts = amazonS3HostDualStack.FindStringSubmatch(endpointURL.Host)
|
||||
if len(parts) > 1 {
|
||||
return parts[1]
|
||||
}
|
||||
|
||||
parts = amazonS3HostHyphen.FindStringSubmatch(endpointURL.Host)
|
||||
if len(parts) > 1 {
|
||||
return parts[1]
|
||||
}
|
||||
|
||||
parts = amazonS3ChinaHost.FindStringSubmatch(endpointURL.Host)
|
||||
if len(parts) > 1 {
|
||||
return parts[1]
|
||||
}
|
||||
|
||||
parts = amazonS3ChinaHostDualStack.FindStringSubmatch(endpointURL.Host)
|
||||
if len(parts) > 1 {
|
||||
return parts[1]
|
||||
}
|
||||
|
||||
parts = amazonS3HostDot.FindStringSubmatch(endpointURL.Host)
|
||||
if len(parts) > 1 {
|
||||
return parts[1]
|
||||
}
|
||||
|
||||
parts = amazonS3HostPrivateLink.FindStringSubmatch(endpointURL.Host)
|
||||
if len(parts) > 1 {
|
||||
return parts[1]
|
||||
}
|
||||
|
||||
return ""
|
||||
}
|
||||
|
||||
// IsAliyunOSSEndpoint - Match if it is exactly Aliyun OSS endpoint.
|
||||
func IsAliyunOSSEndpoint(endpointURL url.URL) bool {
|
||||
return strings.HasSuffix(endpointURL.Host, "aliyuncs.com")
|
||||
}
|
||||
|
||||
// IsAmazonEndpoint - Match if it is exactly Amazon S3 endpoint.
|
||||
func IsAmazonEndpoint(endpointURL url.URL) bool {
|
||||
if endpointURL.Host == "s3-external-1.amazonaws.com" || endpointURL.Host == "s3.amazonaws.com" {
|
||||
return true
|
||||
}
|
||||
return GetRegionFromURL(endpointURL) != ""
|
||||
}
|
||||
|
||||
// IsAmazonGovCloudEndpoint - Match if it is exactly Amazon S3 GovCloud endpoint.
|
||||
func IsAmazonGovCloudEndpoint(endpointURL url.URL) bool {
|
||||
if endpointURL == sentinelURL {
|
||||
return false
|
||||
}
|
||||
return (endpointURL.Host == "s3-us-gov-west-1.amazonaws.com" ||
|
||||
endpointURL.Host == "s3-us-gov-east-1.amazonaws.com" ||
|
||||
IsAmazonFIPSGovCloudEndpoint(endpointURL))
|
||||
}
|
||||
|
||||
// IsAmazonFIPSGovCloudEndpoint - match if the endpoint is FIPS and GovCloud.
|
||||
func IsAmazonFIPSGovCloudEndpoint(endpointURL url.URL) bool {
|
||||
if endpointURL == sentinelURL {
|
||||
return false
|
||||
}
|
||||
return IsAmazonFIPSEndpoint(endpointURL) && strings.Contains(endpointURL.Host, "us-gov-")
|
||||
}
|
||||
|
||||
// IsAmazonFIPSEndpoint - Match if it is exactly Amazon S3 FIPS endpoint.
|
||||
// See https://aws.amazon.com/compliance/fips.
|
||||
func IsAmazonFIPSEndpoint(endpointURL url.URL) bool {
|
||||
if endpointURL == sentinelURL {
|
||||
return false
|
||||
}
|
||||
return strings.HasPrefix(endpointURL.Host, "s3-fips") && strings.HasSuffix(endpointURL.Host, ".amazonaws.com")
|
||||
}
|
||||
|
||||
// IsAmazonPrivateLinkEndpoint - Match if it is exactly Amazon S3 PrivateLink interface endpoint
|
||||
// See https://docs.aws.amazon.com/AmazonS3/latest/userguide/privatelink-interface-endpoints.html.
|
||||
func IsAmazonPrivateLinkEndpoint(endpointURL url.URL) bool {
|
||||
if endpointURL == sentinelURL {
|
||||
return false
|
||||
}
|
||||
return amazonS3HostPrivateLink.MatchString(endpointURL.Host)
|
||||
}
|
||||
|
||||
// IsGoogleEndpoint - Match if it is exactly Google cloud storage endpoint.
|
||||
func IsGoogleEndpoint(endpointURL url.URL) bool {
|
||||
if endpointURL == sentinelURL {
|
||||
return false
|
||||
}
|
||||
return endpointURL.Hostname() == "storage.googleapis.com"
|
||||
}
|
||||
|
||||
// Expects ascii encoded strings - from output of urlEncodePath
|
||||
func percentEncodeSlash(s string) string {
|
||||
return strings.ReplaceAll(s, "/", "%2F")
|
||||
}
|
||||
|
||||
// QueryEncode - encodes query values in their URL encoded form. In
|
||||
// addition to the percent encoding performed by urlEncodePath() used
|
||||
// here, it also percent encodes '/' (forward slash)
|
||||
func QueryEncode(v url.Values) string {
|
||||
if v == nil {
|
||||
return ""
|
||||
}
|
||||
var buf bytes.Buffer
|
||||
keys := make([]string, 0, len(v))
|
||||
for k := range v {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
for _, k := range keys {
|
||||
vs := v[k]
|
||||
prefix := percentEncodeSlash(EncodePath(k)) + "="
|
||||
for _, v := range vs {
|
||||
if buf.Len() > 0 {
|
||||
buf.WriteByte('&')
|
||||
}
|
||||
buf.WriteString(prefix)
|
||||
buf.WriteString(percentEncodeSlash(EncodePath(v)))
|
||||
}
|
||||
}
|
||||
return buf.String()
|
||||
}
|
||||
|
||||
// TagDecode - decodes canonical tag into map of key and value.
|
||||
func TagDecode(ctag string) map[string]string {
|
||||
if ctag == "" {
|
||||
return map[string]string{}
|
||||
}
|
||||
tags := strings.Split(ctag, "&")
|
||||
tagMap := make(map[string]string, len(tags))
|
||||
var err error
|
||||
for _, tag := range tags {
|
||||
kvs := strings.SplitN(tag, "=", 2)
|
||||
if len(kvs) == 0 {
|
||||
return map[string]string{}
|
||||
}
|
||||
if len(kvs) == 1 {
|
||||
return map[string]string{}
|
||||
}
|
||||
tagMap[kvs[0]], err = url.PathUnescape(kvs[1])
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
}
|
||||
return tagMap
|
||||
}
|
||||
|
||||
// TagEncode - encodes tag values in their URL encoded form. In
|
||||
// addition to the percent encoding performed by urlEncodePath() used
|
||||
// here, it also percent encodes '/' (forward slash)
|
||||
func TagEncode(tags map[string]string) string {
|
||||
if tags == nil {
|
||||
return ""
|
||||
}
|
||||
values := url.Values{}
|
||||
for k, v := range tags {
|
||||
values[k] = []string{v}
|
||||
}
|
||||
return QueryEncode(values)
|
||||
}
|
||||
|
||||
// if object matches reserved string, no need to encode them
|
||||
var reservedObjectNames = regexp.MustCompile("^[a-zA-Z0-9-_.~/]+$")
|
||||
|
||||
// EncodePath encode the strings from UTF-8 byte representations to HTML hex escape sequences
|
||||
//
|
||||
// This is necessary since regular url.Parse() and url.Encode() functions do not support UTF-8
|
||||
// non english characters cannot be parsed due to the nature in which url.Encode() is written
|
||||
//
|
||||
// This function on the other hand is a direct replacement for url.Encode() technique to support
|
||||
// pretty much every UTF-8 character.
|
||||
func EncodePath(pathName string) string {
|
||||
if reservedObjectNames.MatchString(pathName) {
|
||||
return pathName
|
||||
}
|
||||
var encodedPathname strings.Builder
|
||||
for _, s := range pathName {
|
||||
if 'A' <= s && s <= 'Z' || 'a' <= s && s <= 'z' || '0' <= s && s <= '9' { // §2.3 Unreserved characters (mark)
|
||||
encodedPathname.WriteRune(s)
|
||||
continue
|
||||
}
|
||||
switch s {
|
||||
case '-', '_', '.', '~', '/': // §2.3 Unreserved characters (mark)
|
||||
encodedPathname.WriteRune(s)
|
||||
continue
|
||||
default:
|
||||
l := utf8.RuneLen(s)
|
||||
if l < 0 {
|
||||
// if utf8 cannot convert return the same string as is
|
||||
return pathName
|
||||
}
|
||||
u := make([]byte, l)
|
||||
utf8.EncodeRune(u, s)
|
||||
for _, r := range u {
|
||||
hex := hex.EncodeToString([]byte{r})
|
||||
encodedPathname.WriteString("%" + strings.ToUpper(hex))
|
||||
}
|
||||
}
|
||||
}
|
||||
return encodedPathname.String()
|
||||
}
|
||||
|
||||
// We support '.' with bucket names but we fallback to using path
|
||||
// style requests instead for such buckets.
|
||||
var (
|
||||
validBucketName = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9\.\-\_\:]{1,61}[A-Za-z0-9]$`)
|
||||
validBucketNameStrict = regexp.MustCompile(`^[a-z0-9][a-z0-9\.\-]{1,61}[a-z0-9]$`)
|
||||
ipAddress = regexp.MustCompile(`^(\d+\.){3}\d+$`)
|
||||
)
|
||||
|
||||
// Common checker for both stricter and basic validation.
|
||||
func checkBucketNameCommon(bucketName string, strict bool) (err error) {
|
||||
if strings.TrimSpace(bucketName) == "" {
|
||||
return errors.New("Bucket name cannot be empty")
|
||||
}
|
||||
if len(bucketName) < 3 {
|
||||
return errors.New("Bucket name cannot be shorter than 3 characters")
|
||||
}
|
||||
if len(bucketName) > 63 {
|
||||
return errors.New("Bucket name cannot be longer than 63 characters")
|
||||
}
|
||||
if ipAddress.MatchString(bucketName) {
|
||||
return errors.New("Bucket name cannot be an ip address")
|
||||
}
|
||||
if strings.Contains(bucketName, "..") || strings.Contains(bucketName, ".-") || strings.Contains(bucketName, "-.") {
|
||||
return errors.New("Bucket name contains invalid characters")
|
||||
}
|
||||
if strict {
|
||||
if !validBucketNameStrict.MatchString(bucketName) {
|
||||
err = errors.New("Bucket name contains invalid characters")
|
||||
}
|
||||
return err
|
||||
}
|
||||
if !validBucketName.MatchString(bucketName) {
|
||||
err = errors.New("Bucket name contains invalid characters")
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
// CheckValidBucketName - checks if we have a valid input bucket name.
|
||||
func CheckValidBucketName(bucketName string) (err error) {
|
||||
return checkBucketNameCommon(bucketName, false)
|
||||
}
|
||||
|
||||
// CheckValidBucketNameStrict - checks if we have a valid input bucket name.
|
||||
// This is a stricter version.
|
||||
// - http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html
|
||||
func CheckValidBucketNameStrict(bucketName string) (err error) {
|
||||
return checkBucketNameCommon(bucketName, true)
|
||||
}
|
||||
|
||||
// CheckValidObjectNamePrefix - checks if we have a valid input object name prefix.
|
||||
// - http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html
|
||||
func CheckValidObjectNamePrefix(objectName string) error {
|
||||
if len(objectName) > 1024 {
|
||||
return errors.New("Object name cannot be longer than 1024 characters")
|
||||
}
|
||||
if !utf8.ValidString(objectName) {
|
||||
return errors.New("Object name with non UTF-8 strings are not supported")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// CheckValidObjectName - checks if we have a valid input object name.
|
||||
// - http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html
|
||||
func CheckValidObjectName(objectName string) error {
|
||||
if strings.TrimSpace(objectName) == "" {
|
||||
return errors.New("Object name cannot be empty")
|
||||
}
|
||||
return CheckValidObjectNamePrefix(objectName)
|
||||
}
|
||||
+195
@@ -0,0 +1,195 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2015-2017 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package set
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
|
||||
"github.com/goccy/go-json"
|
||||
)
|
||||
|
||||
// StringSet - uses map as set of strings.
|
||||
type StringSet map[string]struct{}
|
||||
|
||||
// ToSlice - returns StringSet as string slice.
|
||||
func (set StringSet) ToSlice() []string {
|
||||
keys := make([]string, 0, len(set))
|
||||
for k := range set {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
return keys
|
||||
}
|
||||
|
||||
// IsEmpty - returns whether the set is empty or not.
|
||||
func (set StringSet) IsEmpty() bool {
|
||||
return len(set) == 0
|
||||
}
|
||||
|
||||
// Add - adds string to the set.
|
||||
func (set StringSet) Add(s string) {
|
||||
set[s] = struct{}{}
|
||||
}
|
||||
|
||||
// Remove - removes string in the set. It does nothing if string does not exist in the set.
|
||||
func (set StringSet) Remove(s string) {
|
||||
delete(set, s)
|
||||
}
|
||||
|
||||
// Contains - checks if string is in the set.
|
||||
func (set StringSet) Contains(s string) bool {
|
||||
_, ok := set[s]
|
||||
return ok
|
||||
}
|
||||
|
||||
// FuncMatch - returns new set containing each value who passes match function.
|
||||
// A 'matchFn' should accept element in a set as first argument and
|
||||
// 'matchString' as second argument. The function can do any logic to
|
||||
// compare both the arguments and should return true to accept element in
|
||||
// a set to include in output set else the element is ignored.
|
||||
func (set StringSet) FuncMatch(matchFn func(string, string) bool, matchString string) StringSet {
|
||||
nset := NewStringSet()
|
||||
for k := range set {
|
||||
if matchFn(k, matchString) {
|
||||
nset.Add(k)
|
||||
}
|
||||
}
|
||||
return nset
|
||||
}
|
||||
|
||||
// ApplyFunc - returns new set containing each value processed by 'applyFn'.
|
||||
// A 'applyFn' should accept element in a set as a argument and return
|
||||
// a processed string. The function can do any logic to return a processed
|
||||
// string.
|
||||
func (set StringSet) ApplyFunc(applyFn func(string) string) StringSet {
|
||||
nset := NewStringSet()
|
||||
for k := range set {
|
||||
nset.Add(applyFn(k))
|
||||
}
|
||||
return nset
|
||||
}
|
||||
|
||||
// Equals - checks whether given set is equal to current set or not.
|
||||
func (set StringSet) Equals(sset StringSet) bool {
|
||||
// If length of set is not equal to length of given set, the
|
||||
// set is not equal to given set.
|
||||
if len(set) != len(sset) {
|
||||
return false
|
||||
}
|
||||
|
||||
// As both sets are equal in length, check each elements are equal.
|
||||
for k := range set {
|
||||
if _, ok := sset[k]; !ok {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
// Intersection - returns the intersection with given set as new set.
|
||||
func (set StringSet) Intersection(sset StringSet) StringSet {
|
||||
nset := NewStringSet()
|
||||
for k := range set {
|
||||
if _, ok := sset[k]; ok {
|
||||
nset.Add(k)
|
||||
}
|
||||
}
|
||||
|
||||
return nset
|
||||
}
|
||||
|
||||
// Difference - returns the difference with given set as new set.
|
||||
func (set StringSet) Difference(sset StringSet) StringSet {
|
||||
nset := NewStringSet()
|
||||
for k := range set {
|
||||
if _, ok := sset[k]; !ok {
|
||||
nset.Add(k)
|
||||
}
|
||||
}
|
||||
|
||||
return nset
|
||||
}
|
||||
|
||||
// Union - returns the union with given set as new set.
|
||||
func (set StringSet) Union(sset StringSet) StringSet {
|
||||
nset := NewStringSet()
|
||||
for k := range set {
|
||||
nset.Add(k)
|
||||
}
|
||||
|
||||
for k := range sset {
|
||||
nset.Add(k)
|
||||
}
|
||||
|
||||
return nset
|
||||
}
|
||||
|
||||
// MarshalJSON - converts to JSON data.
|
||||
func (set StringSet) MarshalJSON() ([]byte, error) {
|
||||
return json.Marshal(set.ToSlice())
|
||||
}
|
||||
|
||||
// UnmarshalJSON - parses JSON data and creates new set with it.
|
||||
func (set *StringSet) UnmarshalJSON(data []byte) error {
|
||||
sl := []interface{}{}
|
||||
var err error
|
||||
if err = json.Unmarshal(data, &sl); err == nil {
|
||||
*set = make(StringSet)
|
||||
for _, s := range sl {
|
||||
set.Add(fmt.Sprintf("%v", s))
|
||||
}
|
||||
} else {
|
||||
var s interface{}
|
||||
if err = json.Unmarshal(data, &s); err == nil {
|
||||
*set = make(StringSet)
|
||||
set.Add(fmt.Sprintf("%v", s))
|
||||
}
|
||||
}
|
||||
|
||||
return err
|
||||
}
|
||||
|
||||
// String - returns printable string of the set.
|
||||
func (set StringSet) String() string {
|
||||
return fmt.Sprintf("%s", set.ToSlice())
|
||||
}
|
||||
|
||||
// NewStringSet - creates new string set.
|
||||
func NewStringSet() StringSet {
|
||||
return make(StringSet)
|
||||
}
|
||||
|
||||
// CreateStringSet - creates new string set with given string values.
|
||||
func CreateStringSet(sl ...string) StringSet {
|
||||
set := make(StringSet)
|
||||
for _, k := range sl {
|
||||
set.Add(k)
|
||||
}
|
||||
return set
|
||||
}
|
||||
|
||||
// CopyStringSet - returns copy of given set.
|
||||
func CopyStringSet(set StringSet) StringSet {
|
||||
nset := NewStringSet()
|
||||
for k, v := range set {
|
||||
nset[k] = v
|
||||
}
|
||||
return nset
|
||||
}
|
||||
Generated
Vendored
+224
@@ -0,0 +1,224 @@
|
||||
/*
|
||||
* MinIO Go Library for Amazon S3 Compatible Cloud Storage
|
||||
* Copyright 2022 MinIO, Inc.
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package signer
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// getUnsignedChunkLength - calculates the length of chunk metadata
|
||||
func getUnsignedChunkLength(chunkDataSize int64) int64 {
|
||||
return int64(len(fmt.Sprintf("%x", chunkDataSize))) +
|
||||
crlfLen +
|
||||
chunkDataSize +
|
||||
crlfLen
|
||||
}
|
||||
|
||||
// getUSStreamLength - calculates the length of the overall stream (data + metadata)
|
||||
func getUSStreamLength(dataLen, chunkSize int64, trailers http.Header) int64 {
|
||||
if dataLen <= 0 {
|
||||
return 0
|
||||
}
|
||||
|
||||
chunksCount := int64(dataLen / chunkSize)
|
||||
remainingBytes := int64(dataLen % chunkSize)
|
||||
streamLen := int64(0)
|
||||
streamLen += chunksCount * getUnsignedChunkLength(chunkSize)
|
||||
if remainingBytes > 0 {
|
||||
streamLen += getUnsignedChunkLength(remainingBytes)
|
||||
}
|
||||
streamLen += getUnsignedChunkLength(0)
|
||||
if len(trailers) > 0 {
|
||||
for name, placeholder := range trailers {
|
||||
if len(placeholder) > 0 {
|
||||
streamLen += int64(len(name) + len(trailerKVSeparator) + len(placeholder[0]) + 1)
|
||||
}
|
||||
}
|
||||
streamLen += crlfLen
|
||||
}
|
||||
|
||||
return streamLen
|
||||
}
|
||||
|
||||
// prepareStreamingRequest - prepares a request with appropriate
|
||||
// headers before computing the seed signature.
|
||||
func prepareUSStreamingRequest(req *http.Request, sessionToken string, dataLen int64, timestamp time.Time) {
|
||||
req.TransferEncoding = []string{"aws-chunked"}
|
||||
if sessionToken != "" {
|
||||
req.Header.Set("X-Amz-Security-Token", sessionToken)
|
||||
}
|
||||
|
||||
req.Header.Set("X-Amz-Date", timestamp.Format(iso8601DateFormat))
|
||||
// Set content length with streaming signature for each chunk included.
|
||||
req.ContentLength = getUSStreamLength(dataLen, int64(payloadChunkSize), req.Trailer)
|
||||
}
|
||||
|
||||
// StreamingUSReader implements chunked upload signature as a reader on
|
||||
// top of req.Body's ReaderCloser chunk header;data;... repeat
|
||||
type StreamingUSReader struct {
|
||||
contentLen int64 // Content-Length from req header
|
||||
baseReadCloser io.ReadCloser // underlying io.Reader
|
||||
bytesRead int64 // bytes read from underlying io.Reader
|
||||
buf bytes.Buffer // holds signed chunk
|
||||
chunkBuf []byte // holds raw data read from req Body
|
||||
chunkBufLen int // no. of bytes read so far into chunkBuf
|
||||
done bool // done reading the underlying reader to EOF
|
||||
chunkNum int
|
||||
totalChunks int
|
||||
lastChunkSize int
|
||||
trailer http.Header
|
||||
}
|
||||
|
||||
// writeChunk - signs a chunk read from s.baseReader of chunkLen size.
|
||||
func (s *StreamingUSReader) writeChunk(chunkLen int, addCrLf bool) {
|
||||
s.buf.WriteString(strconv.FormatInt(int64(chunkLen), 16) + "\r\n")
|
||||
|
||||
// Write chunk data into streaming buffer
|
||||
s.buf.Write(s.chunkBuf[:chunkLen])
|
||||
|
||||
// Write the chunk trailer.
|
||||
if addCrLf {
|
||||
s.buf.Write([]byte("\r\n"))
|
||||
}
|
||||
|
||||
// Reset chunkBufLen for next chunk read.
|
||||
s.chunkBufLen = 0
|
||||
s.chunkNum++
|
||||
}
|
||||
|
||||
// addSignedTrailer - adds a trailer with the provided headers,
|
||||
// then signs a chunk and adds it to output.
|
||||
func (s *StreamingUSReader) addTrailer(h http.Header) {
|
||||
olen := len(s.chunkBuf)
|
||||
s.chunkBuf = s.chunkBuf[:0]
|
||||
for k, v := range h {
|
||||
s.chunkBuf = append(s.chunkBuf, []byte(strings.ToLower(k)+trailerKVSeparator+v[0]+"\n")...)
|
||||
}
|
||||
|
||||
s.buf.Write(s.chunkBuf)
|
||||
s.buf.WriteString("\r\n\r\n")
|
||||
|
||||
// Reset chunkBufLen for next chunk read.
|
||||
s.chunkBuf = s.chunkBuf[:olen]
|
||||
s.chunkBufLen = 0
|
||||
s.chunkNum++
|
||||
}
|
||||
|
||||
// StreamingUnsignedV4 - provides chunked upload
|
||||
func StreamingUnsignedV4(req *http.Request, sessionToken string, dataLen int64, reqTime time.Time) *http.Request {
|
||||
// Set headers needed for streaming signature.
|
||||
prepareUSStreamingRequest(req, sessionToken, dataLen, reqTime)
|
||||
|
||||
if req.Body == nil {
|
||||
req.Body = io.NopCloser(bytes.NewReader([]byte("")))
|
||||
}
|
||||
|
||||
stReader := &StreamingUSReader{
|
||||
baseReadCloser: req.Body,
|
||||
chunkBuf: make([]byte, payloadChunkSize),
|
||||
contentLen: dataLen,
|
||||
chunkNum: 1,
|
||||
totalChunks: int((dataLen+payloadChunkSize-1)/payloadChunkSize) + 1,
|
||||
lastChunkSize: int(dataLen % payloadChunkSize),
|
||||
}
|
||||
if len(req.Trailer) > 0 {
|
||||
stReader.trailer = req.Trailer
|
||||
// Remove...
|
||||
req.Trailer = nil
|
||||
}
|
||||
|
||||
req.Body = stReader
|
||||
|
||||
return req
|
||||
}
|
||||
|
||||
// Read - this method performs chunk upload signature providing a
|
||||
// io.Reader interface.
|
||||
func (s *StreamingUSReader) Read(buf []byte) (int, error) {
|
||||
switch {
|
||||
// After the last chunk is read from underlying reader, we
|
||||
// never re-fill s.buf.
|
||||
case s.done:
|
||||
|
||||
// s.buf will be (re-)filled with next chunk when has lesser
|
||||
// bytes than asked for.
|
||||
case s.buf.Len() < len(buf):
|
||||
s.chunkBufLen = 0
|
||||
for {
|
||||
n1, err := s.baseReadCloser.Read(s.chunkBuf[s.chunkBufLen:])
|
||||
// Usually we validate `err` first, but in this case
|
||||
// we are validating n > 0 for the following reasons.
|
||||
//
|
||||
// 1. n > 0, err is one of io.EOF, nil (near end of stream)
|
||||
// A Reader returning a non-zero number of bytes at the end
|
||||
// of the input stream may return either err == EOF or err == nil
|
||||
//
|
||||
// 2. n == 0, err is io.EOF (actual end of stream)
|
||||
//
|
||||
// Callers should always process the n > 0 bytes returned
|
||||
// before considering the error err.
|
||||
if n1 > 0 {
|
||||
s.chunkBufLen += n1
|
||||
s.bytesRead += int64(n1)
|
||||
|
||||
if s.chunkBufLen == payloadChunkSize ||
|
||||
(s.chunkNum == s.totalChunks-1 &&
|
||||
s.chunkBufLen == s.lastChunkSize) {
|
||||
// Sign the chunk and write it to s.buf.
|
||||
s.writeChunk(s.chunkBufLen, true)
|
||||
break
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
if err == io.EOF {
|
||||
// No more data left in baseReader - last chunk.
|
||||
// Done reading the last chunk from baseReader.
|
||||
s.done = true
|
||||
|
||||
// bytes read from baseReader different than
|
||||
// content length provided.
|
||||
if s.bytesRead != s.contentLen {
|
||||
return 0, fmt.Errorf("http: ContentLength=%d with Body length %d", s.contentLen, s.bytesRead)
|
||||
}
|
||||
|
||||
// Sign the chunk and write it to s.buf.
|
||||
s.writeChunk(0, len(s.trailer) == 0)
|
||||
if len(s.trailer) > 0 {
|
||||
// Trailer must be set now.
|
||||
s.addTrailer(s.trailer)
|
||||
}
|
||||
break
|
||||
}
|
||||
return 0, err
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
return s.buf.Read(buf)
|
||||
}
|
||||
|
||||
// Close - this method makes underlying io.ReadCloser's Close method available.
|
||||
func (s *StreamingUSReader) Close() error {
|
||||
return s.baseReadCloser.Close()
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user