GetNotifyNATS reads user_credentials, nkey_seed and tls_handshake_first
and GetNotifyAMQP reads immediate, but none of them were registered in
DefaultNATSKVS/DefaultAMQPKVS or the help schema, so CheckValidKeys
rejected any enable=on target carrying them. Worse, the legacy config
migration wrote exactly these keys - including the env var name
MINIO_NOTIFY_NATS_USER_CREDENTIALS used as a config key, because the
NATSUserCredentials constant doubled as both - so a migrated NATS config
failed validation on every load, and the FetchEnabledTargets fail-fast
then silently disabled all bucket notification targets.
- Register user_credentials/nkey_seed/tls_handshake_first (NATS) and
immediate (AMQP) in the default KVS and help schema; split
NATSUserCredentials into a real config key plus EnvNATSUserCredentials
(all env var names byte-stable)
- Fix legacy migration: SetNotifyNATS writes the proper key;
SetNotifyAMQP no longer writes cfg.Immediate under the internal key
and now carries both immediate and internal
- Tolerate the legacy MINIO_NOTIFY_NATS_USER_CREDENTIALS key written by
pre-fix migrations (NATS-scoped, load path only) with fallback read;
env > user_credentials > legacy key
- Print key names only, never values, in the invalid-keys error of both
CheckValidKeys forms; rejected values can carry credentials
- Add an AST-based audit test asserting parser reads, migration writes
and help entries stay within the registered key set for all ten notify
subsystems, with floor assertions so collector drift fails loudly
- Document (unchanged) FetchEnabledTargets fail-fast and pin it with a
characterization test
Known same-class gap left in place and pinned by the audit's allowlist:
SetNotifyPostgres/SetNotifyMySQL write five unregistered DSN-era keys;
tracked for a follow-up issue.
Closes#39
Co-authored-by: ChatGPT <noreply@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
`go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./...` executed.
`go generate ./...` ran afterwards to keep generated.
The items will be saved per target batch and will
be committed to the queue store when the batch is full
Also, periodically commit the batched items to the queue store
based on configured commit_timeout; default is 30s;
Bonus: compress queue store multi writes
This change uses the updated ldap library in minio/pkg (bumped
up to v3). A new config parameter is added for LDAP configuration to
specify extra user attributes to load from the LDAP server and to store
them as additional claims for the user.
A test is added in sts_handlers.go that shows how to access the LDAP
attributes as a claim.
This is in preparation for adding SSH pubkey authentication to MinIO's SFTP
integration.
i.e., this rule element doesn't apply to DEL markers.
This is a breaking change to how ExpiredObejctDeleteAllVersions
functions today. This is necessary to avoid the following highly probable
footgun scenario in the future.
Scenario:
The user uses tags-based filtering to select an object's time to live(TTL).
The application sometimes deletes objects, too, making its latest
version a DEL marker. The previous implementation skipped tag-based filters
if the newest version was DEL marker, voiding the tag-based TTL. The user is
surprised to find objects that have expired sooner than expected.
* Add DelMarkerExpiration action
This ILM action removes all versions of an object if its
the latest version is a DEL marker.
```xml
<DelMarkerObjectExpiration>
<Days> 10 </Days>
</DelMarkerObjectExpiration>
```
1. Applies only to objects whose,
• The latest version is a DEL marker.
• satisfies the number of days criteria
2. Deletes all versions of this object
3. Associated rule can't have tag-based filtering
Includes,
- New bucket event type for deletion due to DelMarkerExpiration
Create new code paths for multiple subsystems in the code. This will
make maintaing this easier later.
Also introduce bugLogIf() for errors that should not happen in the first
place.
Object names of directory objects qualified for ExpiredObjectAllVersions
must be encoded appropriately before calling on deletePrefix on their
erasure set.
e.g., a directory object and regular objects with overlapping prefixes
could lead to the expiration of regular objects, which is not the
intention of ILM.
```
bucket/dir/ ---> directory object
bucket/dir/obj-1
```
When `bucket/dir/` qualifies for expiration, the current implementation would
remove regular objects under the prefix `bucket/dir/`, in this case,
`bucket/dir/obj-1`.
This patch adds the targetID to the existing notification target metrics
and deprecates the current target metrics which points to the overall
event notification subsystem
use memory for async events when necessary and dequeue them as
needed, for all synchronous events customers must enable
```
MINIO_API_SYNC_EVENTS=on
```
Async events can be lost but is upto to the admin to
decide what they want, we will not create run-away number
of goroutines per event instead we will queue them properly.
Currently the max async workers is set to runtime.GOMAXPROCS(0)
which is more than sufficient in general, but it can be made
configurable in future but may not be needed.
replace io.Discard usage to fix NUMA copy() latencies
On NUMA systems copying from 8K buffer allocated via
io.Discard leads to large latency build-up for every
```
copy(new8kbuf, largebuf)
```
can in-cur upto 1ms worth of latencies on NUMA systems
due to memory sharding across NUMA nodes.
With the current asynchronous behaviour in sending notification events
to the targets, we can't provide guaranteed delivery as the systems
might go for restarts.
For such event-driven use-cases, we can provide an option to enable
synchronous events where the APIs wait until the event is successfully
sent or persisted.
This commit adds 'MINIO_API_SYNC_EVENTS' env which when set to 'on'
will enable sending/persisting events to targets synchronously.
Feng Ruohang