Java Forum / General / December 2006
Java NIO Strategy
mearvk - 07 Dec 2006 18:57 GMT Does anyone have any good strategies for maintaining which read goes to which request? For instance, I am making a file/chat server and obviously clients who have more than a single flow from the server would need some kind of multiplexing scheme. If one client had two file requests, how does one keep the randomness of the reads under control? One might attach a ReadHandler object to the SelectionKey object via the attach() method but these seems contrived (besides I'm already using the attachment for SocketChannel state). Ideas?
Thanks.
EJP - 07 Dec 2006 23:35 GMT > One might attach a ReadHandler object to the SelectionKey object via > the attach() method but these seems contrived (besides I'm already > using the attachment for SocketChannel state). The attachment is generally used as a Session object. It can contain the SocketChannel state (what state would that be?), information about the user, information about the current transaction.
Alternatively you can use the SelectionKey as a key into a Map<SelectionKey,Session>, but this seems contrived to me compared to using the key attachment.
mearvk - 08 Dec 2006 00:43 GMT > The attachment is generally used as a Session object. It can contain the > SocketChannel state (what state would that be?), information about the [quoted text clipped - 3 lines] > Map<SelectionKey,Session>, but this seems contrived to me compared to > using the key attachment. I keep track of things like what state the server thinks the client is in, in a SocketChannelState object. For instance, my clients have a protocol to login. They cannot perform any meaningful commands until they have performed the login protocol. So, the best quick solution I could come up with is to maintain state for each SelectionKey via the attach() method. At first I thought this might be problematic because I was worried that the key's state would also get removed on the iterator.remove() call, but I have found this technique workable. However, for multiple logical flows (encryption to different endpoints for instance) over the same physical flow, I am quickly realising this requires some heavier-duty stateful objects. The Map is worth considering as I will ultimately have to build more state into my program.
Anyways, for all the hurrahs about NIO I am finding it rather cumbersome. If you have any good strategies for these kinds of issues, feel free to let me know! :-)
Thanks for your reply,
Mearvk
Wesley Hall - 08 Dec 2006 01:28 GMT >> The attachment is generally used as a Session object. It can contain the >> SocketChannel state (what state would that be?), information about the [quoted text clipped - 25 lines] > > Mearvk Mearvk,
You are right, the NIO libraries are not simple to work with, SSL is especially troublesome.
To solve your problem, you may want to consider creating a 'Session' object and making your connection state value a field within that 'Session' object. This will allow you to store other required values within this object.
Karl Uppiano - 08 Dec 2006 04:28 GMT >>> The attachment is generally used as a Session object. It can contain the >>> SocketChannel state (what state would that be?), information about the [quoted text clipped - 35 lines] > 'Session' object. This will allow you to store other required values > within this object. I keep everything about the client in the attachment, including the callback (event listener) to notify the client of incoming data. I don't have to look up, or switch or run any conditional logic. I simply execute -- Bam! I think NIO is a beautiful thing.
wesley.hall@gmail.com - 08 Dec 2006 12:31 GMT > >>> The attachment is generally used as a Session object. It can contain the > >>> SocketChannel state (what state would that be?), information about the [quoted text clipped - 40 lines] > up, or switch or run any conditional logic. I simply execute -- Bam! I think > NIO is a beautiful thing. As long as you can handle 'part messages' then this is a nice approach.
The problem with having a heavy 'attachment' on selection keys is that if the key doesn't wake up, that attachment is held indefinately. You have no oppurtunity to close the connection and no oppurtunity to kill the key that is holding on to your attachement. It wont be GCed.
An NIO application that has heavy key attachments and an unreliable upstream network has the potential to leak memory like a sieve. Something to be aware of.
Karl Uppiano - 08 Dec 2006 21:14 GMT >> >>> The attachment is generally used as a Session object. It can contain >> >>> the [quoted text clipped - 57 lines] > upstream network has the potential to leak memory like a sieve. > Something to be aware of. My particular application is TELNET terminal applications (TN3270/E, TN5250, Unisys, VT, etc.). So the "client" is actually a TN decoder. Whenever the channel receives data, the selector wakes up and calls the attached TELNET decoder, filling in a buffer with decoded data. When a packet containing a TELNET EOR (end of record) is received, the buffer is forwarded to the terminal for further processing and display. When client needs to send data, we "wake up" the selector to gain access to the selector thread, to encode the data and send it off (note that we do not use async NIO for transmit - it does not seem worth the trouble for what we have to send. There is no significant delay or wait time for the transmit buffer to drain out). Then it goes back to monitoring connections for received data. If the connection is closed by the host, the selector wakes up. Or we can wake up the selector and close it on our end. This is a commercial product in a very high volume server application, and we have not had any problem with memory or resource leaks. I cannot remember all of the details of our application at the moment, but we do have inactivity timeouts to reclaim unresponsive connections.
EJP - 08 Dec 2006 23:49 GMT > The problem with having a heavy 'attachment' on selection keys is that > if the key doesn't wake up, that attachment is held indefinately. You [quoted text clipped - 4 lines] > upstream network has the potential to leak memory like a sieve. > Something to be aware of. Any serious NIO application should use a timed select and have an idle process that is run when the select times out with no ready keys. The idle process should scan the registered key set for channels which haven't done anything for a while, using last-read/last-write timers held in the Session attachment, and take application action to close these connections. This is true both for channels which haven't sent anything for too long, however long that may be, indicating that the session has timed out, and channels to which you haven't been able to write for too long, indicating that the peer is stalled.
Daniel Dyer - 09 Dec 2006 00:14 GMT >> The problem with having a heavy 'attachment' on selection keys is that >> if the key doesn't wake up, that attachment is held indefinately. You [quoted text clipped - 13 lines] > session has timed out, and channels to which you haven't been able to > write for too long, indicating that the peer is stalled. If the OP wants more information on this particular issue, this article identifies the problem and discusses possible approaches:
"Why SelectionKey.attach() is evil" http://weblogs.java.net/blog/jfarcand/archive/2006/06/tricks_and_tips.html
The author's other articles on NIO might also worth reading:
"Why you must handle OP_WRITE" http://weblogs.java.net/blog/jfarcand/archive/2006/05/tricks_and_tips_1.html
"To Thread or Not to Thread" http://weblogs.java.net/blog/jfarcand/archive/2006/07/tricks_and_tips_3.html
"Meet the Selectors" http://weblogs.java.net/blog/jfarcand/archive/2006/07/tricks_and_tips_4.html
Dan.
 Signature Daniel Dyer http://www.uncommons.org
mearvk - 09 Dec 2006 05:24 GMT EJP - 10 Dec 2006 03:28 GMT > "Why SelectionKey.attach() is evil" > http://weblogs.java.net/blog/jfarcand/archive/2006/06/tricks_and_tips.html I've read all those and I don't find them at all convincing. Every accepted connection represents a session, and the session state, whatever it may be, has to be held somewhere. Why not in the attachment? And what is it that makes using the key attachment for the session state 'the devil'? And the part about starting a second selector in the same thread to complete a partial read, introducing another block, is sheer nonsense. *Not* the way to implement a highly scalable server, thanks. You already have a Selector: use it! let it trigger when there is more data, and in the meantime let it handle all the other channels!
What you don't need with NIO is a large read buffer: you can do with quite a small one, e.g. 1k, if you get the strategy for partial reads and writes right.
His piece on OP_WRITE has this gem:
while ( bb.hasRemaining() ) { int len = socketChannel.write(bb); if (len < 0){ throw new EOFException(); } }
'This code will works most of the time....until the Selector on which the SocketChannel has been registered is exhausted, e.g the Selector isn't able to let the socketChannel flush the content of the ByteBuffer.'
Now (a) write() never returns -1, so what's the test for? (b) Selectors don't 'get exhausted', and the rest of the last sentence is nonsense. He explains it further in a response to a comment: 'I means the Selector is not able to let the socketChannel write its buffer. When this happens, the socketChannel.write(bb) will return a value of 0, meaning no bytes were written.' And this is *still* nonsense. Selectors don't prevent channels from doing anything with their buffers.
What really happens is that the socket send buffer fills up if the reader is slow, and *this* causes write() to return 0. And once again he uses a temporary selector to 'solve' this problem. And once again this is sheer nonsense. He 'clarifies' this two days later in response to another comment which states the case correctly, blaming tinking in French/writing in English for the error. Not a plausible explanation.
What you should do is:
while ( bb.hasRemaining() ) { int len = socketChannel.write(bb); if (len == 0){ break; } }
and then you do the bb.compact(), and then if there are still bytes unwritten you register for OP_WRITE, otherwise you deregister it. Once again you let the original Selector do the work so it can handle other threads in the meantime.
Part IV talks about using multiple threads and multiple selectors so as 'not to overload the main Selector'. What does this mean? Overload the thread it's running in? There can't possibly be any benefit unless there are multiple processors and the threads each run in a different processor. It's just a needless complication otherwise, and I'd like to see some figures that prove it can be a genuine benefit even in the multi-processor case. Also in this part he is now recommending using the key attachment, contradicting what he said in part I.
Very curious set of blogs.
mearvk - 10 Dec 2006 05:07 GMT EJP thank you for your comments.
Obviously NIO, to many, is a bit confusing. Do you have any links or references you can provide the rest of us? I'm sure we would appreciate it.
Thanks,
Mearvk
EJP - 10 Dec 2006 09:45 GMT > EJP thank you for your comments. > > Obviously NIO, to many, is a bit confusing. Do you have any links or > references you can provide the rest of us? I'm sure we would appreciate > it. <plug> http://www.telekinesis.com.au/wipv3_6/FundamentalNetworkingInJava.A21 </plug>
mearvk - 10 Dec 2006 23:02 GMT Shameless plug... I love it! :-)
Your book has great reviews on Amazon.com, so I decided to pick it up.
Thanks,
Mearvk
mearvk - 11 Dec 2006 20:28 GMT EJP,
Does it make any kind of sense do have multiple SocketChannels chained to a single client socket (somehow) in order to handle buffering multiple logical flows from that single client? I'm not sure if this is even possible. However, basically I need a sound strategy for multiplexing client SocketChannel reads. For instance, a client may request a file, then another file, then a stock quote. All requests may be received at the client concurrently. What is the best way to sort each request to it intended destination (ByteBuffer)?
Thanks,
Mearvk
EJP - 11 Dec 2006 22:28 GMT > Does it make any kind of sense do have multiple SocketChannels chained > to a single client socket (somehow) in order to handle buffering > multiple logical flows from that single client? No.
> I'm not sure if this is even possible. No.
> However, basically I need a sound strategy for > multiplexing client SocketChannel reads. For instance, a client may > request a file, then another file, then a stock quote. All requests may > be received at the client concurrently. What is the best way to sort > each request to it intended destination (ByteBuffer)? They can only arrive at the server sequentially, unless the client opens multiple connections. Maybe that's what you want to do?
mearvk - 12 Dec 2006 00:00 GMT So basically if I want to demux concurrent (multiple overlapping readFromSocketChannel calls) incoming flows on my client, the best way is to do it is to create multiple SocketChannels initially? I wouldn't be against this but I want a sanity check on this thinking...
To illustrate more clearly:
The client requests 3 files from the server using a single SocketChannel. These files are large and therefore require several trips each to readFromSocketChannel. Now since the SocketChannel has only one associated SelectionKey how do I match the bytes to the proper ByteBuffers (and eventually to their File objects)? Unless I preface each Server write with a header like (or some such scheme):
"RequestID:123456789"
and use these to reference into a Vector<ByteBuffer> (starting to get complicated!) how am I to demux on the client sans multiple SocketChannels?
Thanks,
Mearvk
EJP - 12 Dec 2006 05:31 GMT > So basically if I want to demux concurrent (multiple overlapping > readFromSocketChannel calls) incoming flows on my client, the best way [quoted text clipped - 11 lines] > > "RequestID:123456789" Yep, that's what you would have to do. You'll also have to write length words into each chunk that you transmit so you can pick apart the responses at the receiving end.
I would definitely do this via multiple sockets myself.
mearvk - 11 Dec 2006 21:20 GMT [Quote from: http://weblogs.java.net/blog/jfarcand/archive/2006/06/tricks_and_tips.html]
At this stage, socketChannel is ready to read bytes. Hence you invoke socketChannel.read(byteBuffer), and you find that you haven't read all the bytes from the socket (or you are ready to handle the next request), so you decide to register the SelectionKey back to the Selector by doing:
selectionKey.interestOps(selectionKey.interestOps() | SelectionKey.OP_READ);
and...and...and do something like:
selectionKey.attach(...)
Boum...the little ... is where the devil is hiding! What you are attaching to the SelectionKey is very dangerous, because there is some probability that your SelectionKey might never return to a ready-operation state, leaving the SelectionKey and its evil attachment forever inside the Selector keys set.
[/Quote]
Does this make sense? If you explicitly tell your program to re-insert the SelectionKey back into the key set and we know it has more data to be read (per the premise), then unless your server is completely pegged, wouldn't it *always* return to a ready-operation state?
Also, instead of:
[Quote]
while ( socketChannel.isOpen() && (count = socketChannel.read(byteBuffer))> -1)){ // do something }
[/Quote]
Could you do:
if((count = socketChannel.read(byteBuffer))> -1) { //read into temp buffer } else { //forward buffer }
So that your time in the readFromSocketChannel method is shorter/fairer? This assumes that the key gets placed back into the Selector key set and eventually (depending on load)gets re-handled promptly.
Mearvk
EJP - 11 Dec 2006 22:34 GMT > [Quote from: > http://weblogs.java.net/blog/jfarcand/archive/2006/06/tricks_and_tips.html] [quoted text clipped - 7 lines] > selectionKey.interestOps(selectionKey.interestOps() | > SelectionKey.OP_READ); Yet another thing I don't understand in these blogs. You should only have gotten here if you were already registered for OP_READ. Just stay that way!
> Boum...the little ... is where the devil is hiding! What you are > attaching to the SelectionKey is very dangerous, because there is some > probability that your SelectionKey might never return to a > ready-operation state, leaving the SelectionKey and its evil attachment > forever inside the Selector keys set. Rubbish. This can only happen if you don't do the idle processing I described in an earlier posting. *That's* the devil.
> Does this make sense? If you explicitly tell your program to re-insert > the SelectionKey back into the key set and we know it has more data to > be read (per the premise), then unless your server is completely > pegged, wouldn't it *always* return to a ready-operation state? Only if more data arrives.
The 'premiss' is meaningless: 'At this stage, socketChannel is ready to read bytes. Hence you invoke socketChannel.read(byteBuffer), and you find that you haven't read all the bytes from the socket (or you are ready to handle the next request)'. How can you find you haven't read all the bytes from the socket, except by trying another read? And this case even if it existed is logically very different from the case where you are ready to handle the next request.
> Could you do: > [quoted text clipped - 11 lines] > Selector key set and eventually (depending on load)gets re-handled > promptly. I agree and this is what I do except that I don't do all this key manipulation. Also I check for 0 and -1 separately as they are very different cases.
I register and de-register as follows (assuming this is a server):
(a) when I get an OP_ACCEPT and accept a channel I register it for OP_READ.
(b) When I've had enough read events and read enough data to constitute a complete request (an interesting problem in itself) I deregister for OP_READ and pass the request off for processing. If I get EOF instead I close the channel, physically and logically.
(c) When I get the response back from wherever it was processed I attempt a write. If this doesn't succeed completely I register OP_WRITE. Any time I get OP_WRITE and the write succeeds completely I deregister OP_WRITE and register OP_READ.
(d) I do the reads and writes in a single attempt without looping, for better fairness between channels.
(e) In the idle loop, if I find a channel that has been registered for OP_WRITE for too long I close it physically and logically and abort the transaction internally. If I find a channel that has been registered for OP_READ for a long time I might time out the connection, depending on the application. If I find a channel that hasn't been registered for *anything* for a long time, it means that some transaction is still in progress and I might want to inquire into why it is taking so long.
(f) If the transactions are such that there are done in-line rather than in separate worker threads, i.e. inline at the OP_READ site, when I get the response I don't write it straight away, I register OP_WRITE on the channel and deregister OP_READ, and let the reply be written out on the next iteration of the Selector. Again, this promotes fairness among channels.
The reason for deregistering OP_READ in each case above is that you usually can't logically handle another request from the same channel until you've completed the previous one and written the reply. So you should quench that channel. Otherwise you have to read the new data, and put it somewhere, which takes memory. Better to stop reading and, eventually, stall the sender, by closing the TCP window. It's a bit like the principle of letting passengers get off the bus before the new ones get on.
The really scary thing about these blogs is that this guy apparently works for Sun Microsystems. Here's another gem I noticed:
while (bb.hasRemaining()) { int len = ch.write(bb); if (len < 0) throw new EOFException(); }
etc. I've already commented that write() never returns a negative result. What fascinates me now is the concept of throwing an EOFException when *writing*. This seems to have been lifted holus-bolus from the old NIO tutorial code, which was obviously cut-and-pasted from the read code, and which was corrected at my request last year as being meaningless.
mearvk - 12 Dec 2006 00:35 GMT [Quote]
> Does this make sense? If you explicitly tell your program to re-insert > the SelectionKey back into the key set and we know it has more data to > be read (per the premise), then unless your server is completely > pegged, wouldn't it *always* return to a ready-operation state? Only if more data arrives.
[/Quote]
So calling interestOps 'resets' the SocketChannel's awareness of its buffer? I'm a bit confused on this point. If we omit the interestOps call do we have a working system? Putting a partially-read SocketChannel back into the Selector's awareness *should* cause the Selector.select() call to return at least 1, even without new data arriving, should it not?
[Quote]
(b) When I've had enough read events and read enough data to constitute
a complete request (an interesting problem in itself) I deregister for OP_READ and pass the request off for processing. If I get EOF instead I
close the channel, physically and logically.
(c) When I get the response back from wherever it was processed I attempt a write. If this doesn't succeed completely I register OP_WRITE. Any time I get OP_WRITE and the write succeeds completely I deregister OP_WRITE and register OP_READ.
[/Quote]
So basically what I'd like to know is how you actually implemented this? If you implement it in the readFromSocketChannel call, then your whole program blocks while waiting for the processing call to return, right? So do you start a new thread and use a notification system, or what?
Anyways, thanks again.
Mearvk
EJP - 12 Dec 2006 05:54 GMT > So calling interestOps 'resets' the SocketChannel's awareness of its > buffer? No. I was assuming a complete read of the available data. I should have said something like 'only if and when more data is or becomes available in the socket receive buffer'.
However the situation we are both quoting from in the blog is completely meaningless because you *can't* 'know it has more data to be read'.
> I'm a bit confused on this point. If we omit the interestOps > call do we have a working system? Putting a partially-read > SocketChannel back into the Selector's awareness *should* cause the > Selector.select() call to return at least 1, even without new data > arriving, should it not? Yes, putting a partially-read socket channel back into the Selector's awareness does indeed cause it to be selected again. But so would leaving it alone, unless the first thing the read code does is deregister the key for OP_READ - but why would it do that?
The entire situation is bogus. You accept a connection; you register it for OP_READ. OP_READ fires for the channel; you do a read; you get a read count; you look at the buffer. If it now contains a complete request you process it. Otherwise you just return to the ready-key-processing loop, process more keys, and eventually you return to the select(); eventually, or immediately, or never, it fires another OP_READ for this channel, so you do another read, appending to the same buffer.
You don't have to register/re-register for this second OP_READ because you're *already* registered for OP_READ: that's why you did the *first* read.
It's all much simpler than presented in that futile blog. *When you have the complete request* is the time to deregister for OP_READ. The less you fiddle with things unnecessarily the better, surely?
> So basically what I'd like to know is how you actually implemented > this? If you implement it in the readFromSocketChannel call, then your > whole program blocks while waiting for the processing call to return, > right? Of course not. That would be just as ridiculous as the original suggestion to use a second selector and block eveybody else on that. NIO is for concurrency and scalability, and blocking on anything except a select() is strictly out.
> So do you start a new thread and use a notification system ...? Of course. In systems where executing the transaction can block (e.g. because of database contention), you have worker threads and all that jazz. However I do have one system where my server is the only user of the database and it all fits into memory, so I just do the database operations in line in the selecting thread. This was kind of forced on me and not my ideal choice, but in practice it seems to work well enough.
EJP - 12 Dec 2006 06:56 GMT >> "Why SelectionKey.attach() is evil" >> http://weblogs.java.net/blog/jfarcand/archive/2006/06/tricks_and_tips.html To save further discussion I'll deal with this blog here in its entirety.
> if ((key.readyOps() & SelectionKey.OP_ACCEPT) == > SelectionKey.OP_ACCEPT){ [quoted text clipped - 10 lines] > // do something > } So far so good. Obviously the channel must be registered for OP_READ otherwise we would never have got here.
> Well, the scary part is the // do something. > Gold Candidate for a Memory Leak (GCML) > At this stage, socketChannel is ready to read bytes. Hence you invoke > socketChannel.read(byteBuffer), and you find that you haven't read all > the bytes from the socket
How can you possibly find that? or does he mean that you find you haven't read the entire request?
> so you decide to register the SelectionKey back to the Selector by > doing:
> selectionKey.interestOps( > selectionKey.interestOps() | SelectionKey.OP_READ); What for? OP_READ IS ALREADY REGISTERED! How else did we get here?
> and...and...and do something like: > > selectionKey.attach(...) > > Boum...the little ... is where the devil is hiding! etc etc etc, all the stuff about how selection keys with attachments that never fire events are memory leaks. Well, such a selection key is *itself* a memory leak, and you have to scan for that case anyway. So when you scan the selector's keyset for idle keys, you've just solved the entire imaginary problem.
You also have to maintain session context anyway, and you have to hold this somewhere, and the selection-key attachment is as good a place as any. Better IMHO.
> How do I retrieve the SocketChannel if I don't attach it to my framework object.
A non-problem. SelectionKey.channel().
> How do I deal with incomplete socketChannel read.
> When you do socketChannel.read(), you can never predict when all bytes > are read from the socket buffer. Most of the time, you will have to
> register the SelectionKey back to the Selector, waiting for more bytes > to be available. Why? It's already registered.
> In that case, you will most probably attach the incomplete ByteBuffer > to the SelectionKey, and continue adding bytes to it once the > SelectionKey is ready. I would already have attached a session object when I accepted the connection, and the ByteBuffer would be part of that. Possibly two, if I want to maintain a read buffer and a write buffer.
> Instead, I would recommend you register the > SelectionKey to a temporary Selector (I will blog about this trick in [quoted text clipped - 11 lines] > .register(readSelector,SelectionKey.OP_READ); > tmpKey.interestOps(tmpKey.interestOps() | SelectionKey.OP_READ);
The last line is redundant.
> int code = readSelector.select(readTimeout); At this point the entire application, with its 10,000 connections whose liveness we are so concerned about, has been stalled for up to 15 seconds waiting for more data from just one channel. This suggestion is just plain stupid.
> tmpKey.interestOps( > tmpKey.interestOps() & (~SelectionKey.OP_READ)); What's this for?
He doesn't show why this entire extra Selector isn't a memory leak in itself. I suppose the selector factory is doing something about that, in which case this line might be important. Or not. I can imagine the selector factory taking care of that itself actually.
But I certainly wouldn't be doing any of this.
> With this trick, you don't need to attach anything to the SelectionKey. With this trick you have just lost all concurrency and scalability. Your server has just dedicated itself to a single client for 15 seconds.
> So here you gonna need to decide based on your use of NIO: do you want a dormant ByteBuffer attached to a SelectionKey or a Thread blocking for readTimeout.
Clearly a non-decision. We are using NIO because we want scalability.
> In Grizzly, both approaches can be configured, but by default the thread will block for 15 seconds and cancel the SelectionKey if the client isn't doing anything.
Remind me to avoid Grizzly in that case.
> You can configure Grizzly to attach the ByteBuffer if you really like to use memory :-) . We did try on slow network, with broken client, etc., and blocking a Thread scale better than having a dormant ByteBuffer,
This claim is not credible.
> [he then goes on to describe an architecture where you start more threads to handle these incomplete reads]
Once again losing scability in favour of more threads depending on load.
> Wow that one was very long. Agree, disagree?..... I disagree completely. NIO is just not that complicated to use well. Considering this effort comes from within Sun it is really quite shocking that it exhibits little if any understanding of what NIO really does or how to use it.
mearvk - 12 Dec 2006 07:29 GMT I appreciate your help. See you around the boards.
Mearvk
mearvk - 08 Dec 2006 19:11 GMT Karl can I get you to expand your implementation for those of us who haven't mastered NIO?
Thanks,
Mearvk
Karl Uppiano - 08 Dec 2006 21:15 GMT > Karl can I get you to expand your implementation for those of us who > haven't mastered NIO? > > Thanks, Sure, I responded to the sibling of this post. I can elaborate more if you want. Just ask. - Karl
Free MagazinesGet these publications absolutely FREE for up to 12 months. There are no hidden fees and no obligation. Simply choose a title, complete the application form and submit it. Read more ...
|
|
|