How to concurrently write data in Java without locking?

How to implement a multi-threaded program in which each thread needs to write/output some data (in any order of course) but without the locking overhead that is involved with files or a particular data structure. Is each thread writing to a separate file a feasible option? (wrt to locking overhead) Can we use System.out.print or BufferWriter instead of files? Or is there any other elegant solution?

asked May 3, 2015 at 2:42 Outflanker Outflanker 133 1 1 gold badge 1 1 silver badge 3 3 bronze badges

Another thing to note: the data written by each thread is less than 100 bytes, however the number of threads can be close to a hundred.

Commented May 3, 2015 at 2:47 Commented May 3, 2015 at 3:04

What exactly do you mean by "locking overhead that is involved with files". Maybe you mean blocking characteristic of file operations? In that case, producer-consumer pattern could help.

Commented May 3, 2015 at 8:08

1 Answer 1

If you have multiple threads generating data independently from each other, writing to separate files may be indeed a good option. There will still be some locking under the hood since the Operating System needs to keep the filesystem consistent, but it should be better than writing to a single file with manual locking or writing to standard output (which has internal synchronization). Similarily if you have access to some online storage which you can access via the network, you could send the data independently from each thread.

But there is a big "but". If you write out a lot of data (more than several MB/s, depending on machine specs), your file system may not be able to keep up. In such a situation, the bottleneck will be your file system and not locks in the app. If this happens, you might actually be better off with a single file and locking since concurrent writes to multiple locations may put more load on the filesystem and disk than writing at a single spot. So it all depends and any number of concurrent writes between one and many may be optimal. Your best bet would be to make the number of concurrent writes parametrizable and then to compare the final performance of the application for several different values.

answered May 3, 2015 at 10:07 Michał Kosmulski Michał Kosmulski 3,534 20 20 silver badges 18 18 bronze badges

Related

Hot Network Questions

Subscribe to RSS

Question feed

To subscribe to this RSS feed, copy and paste this URL into your RSS reader.

Site design / logo © 2024 Stack Exchange Inc; user contributions licensed under CC BY-SA . rev 2024.9.4.14806