conduit-map of lazy seqs

[ronslow]

I have a function which operates on lazy seqs, such that when using map, I need a doall:

(map pipeline-fn lazy-seq) => only the first is sent through pipeline

(doall (map pipeline-fn lazy-seq)) => the whole seq is sent through pipeline

When I substitute conduit-map, only the first in the seq is fed to the pipeline, whether with or without doall:

(doall (conduit-map pipeline-proc lazy-seq)) => only the first is sent through the pipeline

What is the correct way to send a lazy seq through a pipeline using conduit?

Thanks

Robert

[jduey]

That looks like the correct usage. What's the context?

I hacked this together:

(ns conduit.temp
(:use conduit.core))

(defn li [i](lazy-seq
%28cons i
%28do
%28println :i i%29
%28li %28+ 1 i%29%29%29%29))

(def ip (a-arr (fn [i](+ 1 i))))

(def cp (conduit-map ip (li 4)))

And cp got assigned a lazy list.

[ronslow]

Jim
Thanks.
You are right to ask about the context. the lazy seq in question is
generated by the clojureql library, so there is some weird interaction
happening here.
I think I have solved it by a hack:

(doall (map identity (conduit-map pipeline lazy-seq)))

I have another question.
I have a proc which needs to "multiply" the stream i.e. generate a sequence
of outputs in response to 1 input.
What's the correct way to do this?

Thanks for your help, Jim. I love the library - which has hugely simplified
the topology of my app!

Robert

-----Original Message-----
From: Jim Duey
Sent: Tuesday, September 20, 2011 6:42 PM
To: ronslow
Subject: Re: [conduit] conduit-map of lazy seqs (#2)

That looks like the correct usage. What's the context?

I hacked this together:

(ns conduit.temp
(:use conduit.core))

(defn li [i](lazy-seq
%28cons i
%28do
%28println :i i%29
%28li %28+ 1 i%29%29%29%29))

(def ip (a-arr (fn [i](+ 1 i))))

(def cp (conduit-map ip (li 4)))

And cp got assigned a lazy list.

Reply to this email directly or view it on GitHub:
#2 (comment)

[jduey]

Thanks for the compliment. Conduit is some of the best code I've written in Clojure and it's gratifying to hear it helps people.

That hack is puzzling to me. I'd be interested in seeing more of the context and maybe I could figure out what is going on.

'multiplying' results breaks things. :) As logical as it appears at first glance to be able to do something like that, it breaks the way streams are modeled in conduit. The reason is given in the paper by John Hughes called "Generalizing monads to arrows".

I'd say try thinking about the problem different. If you'd like to describe your problem a little bit, I'd be glad to help.

[ronslow]

Jim
The problem is this. I am processing status updates from Facebook for a
number of Facebook users.

I pick up the user credentials from a database and drive them as a seq
through a Conduit pipeline. The first component in the pipeline reads status
updates from Facebook for the user. That results in multiple status updates
per user.

Now what I would like to do is to multiply the stream and drive each status
update, with the corresponding user credentials, as a stream through the
rest of the pipeline. I would prefer not to have the restriction that
downstream components in the pipeline must be capable of processing multiple
status updates. Is this legitimate using arrows? Is there some other
topology in Conduit which would permit me to insert downstream components
which process one status update at a time.

Thanks

Robert

-----Original Message-----
From: Jim Duey
Sent: Wednesday, September 21, 2011 3:57 PM
To: ronslow
Subject: Re: [conduit] conduit-map of lazy seqs (#2)

Thanks for the compliment. Conduit is some of the best code I've written in
Clojure and it's gratifying to hear it helps people.

That hack is puzzling to me. I'd be interested in seeing more of the context
and maybe I could figure out what is going on.

'multiplying' results breaks things. :) As logical as it appears at first
glance to be able to do something like that, it breaks the way streams are
modeled in conduit. The reason is given in the paper by John Hughes called
"Generalizing monads to arrows".

I'd say try thinking about the problem different. If you'd like to describe
your problem a little bit, I'd be glad to help.

Reply to this email directly or view it on GitHub:
#2 (comment)

[jduey]

Hey Robert,

I don't see the response I sent you about using disperse. Did you get it?

Also, did you see the other issue about the rewrite of conduit?

Jim

[ronslow]

Jim,
Didn't get it! Could you resend?
Saw about the rewrite of conduit which I will get onto.
Robert

-----Original Message-----
From: Jim Duey
Sent: Sunday, September 25, 2011 2:28 AM
To: ronslow
Subject: Re: [conduit] conduit-map of lazy seqs (#2)

Hey Robert,

I don't see the response I sent you about using disperse. Did you get it?

Also, did you see the other issue about the rewrite of conduit?

Jim

Reply to this email directly or view it on GitHub:
#2 (comment)

[ronslow]

Jim

So the "for" statement in add-statuses is "multiplying" the stream in the
way that I had envisaged. Yes. Exactly what I need.

The "disperse" in your example is a nice extra feature to do parallel
processing of all status updates at once, I guess. Is that right?

Robert

-----Original Message-----
From: Jim Duey
Sent: Sunday, September 25, 2011 10:28 PM
To: ronslow
Subject: Re: [conduit] conduit-map of lazy seqs (#2)

Good morning Robert.

I'm to make a couple of assumptions:

That credentials are given as maps and
That processing of each status update is independent

With that, you could do something like:

(def-arr add-statuses [creds](for [status %28get-statuses cred%29]
%28assoc creds :status status%29))

(def-arr process-status [status]
; do something with a single status
)

(def process-account
(a-comp add-statuses
(disperse process-status)))

(conduit-map process-account credentials-list)

Is that all clear? If not, I can explain more fully.

Good luck,
Jim

Reply to this email directly or view it on GitHub:
#2 (comment)

[jduey]

Conceptually, it's in parallel. But what actually happens is each value is sent through the proc one at a time.

[ronslow]

Jim
I'm getting there. But ..

(defn p [param](let [p1 %28println param%29]
param)
)

(with-arrow conduit (def pipeline (a-arr p) )

(conduit-map pipeline (range 5))

Only 0 is printed. Why?

Robert

-----Original Message-----
From: Jim Duey
Sent: Monday, September 26, 2011 2:31 PM
To: ronslow
Subject: Re: [conduit] conduit-map of lazy seqs (#2)

Conceptually, it's in parallel. But what actually happens is each value is
sent through the proc one at a time.

Reply to this email directly or view it on GitHub:
#2 (comment)

[jduey]

I don't know. It worked for me when I pasted that into a REPL.

I did have to add another ) though.